Automated Maintenance of External Tables in Database Systems

ABSTRACT

Systems, methods, and devices for automated maintenance of external tables in database systems are disclosed. A method includes receiving, by a database platform, read access to content in an external data storage platform that is separate from the database platform. The method includes defining an external table based on the content in the external data storage platform. The method includes connecting the database platform to the external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes generating metadata for the external table, the metadata comprising information about data stored in the external table. The method includes receiving a notification that a modification has been made to the content in the external data storage platform, the modification comprising one or more of an addition of a file, a deletion of a file, or an update to a file in a source location for the external table. The method includes refreshing the metadata for the external table in response to the modification being made to the content in the external data storage platform.

CROSS-REFERENCE TO RELATED APPLICATIONS

Not Applicable.

TECHNICAL FIELD

The present disclosure relates to databases and more particularlyrelates to external tables in database systems.

BACKGROUND

Databases are widely used for data storage and access in computingapplications. A goal of database storage is to provide enormous sums ofinformation in an organized manner so that it can be accessed, managed,and updated. In a database, data may be organized into rows, columns,and tables. Different database storage systems may be used for storingdifferent types of content, such as bibliographic, full text, numeric,and/or image content. Further, in computing, different database systemsmay be classified according to the organization approach of thedatabase. There are many different types of databases, includingrelational databases, distributed databases, cloud databases,object-oriented and others.

Databases are used by various entities and companies for storinginformation that may need to be accessed or analyzed. In an example, aretail company may store a listing of all sales transactions in adatabase. The database may include information about when a transactionoccurred, where it occurred, a total cost of the transaction, anidentifier and/or description of all items that were purchased in thetransaction, and so forth. The same retail company may also store, forexample, employee information in that same database that might includeemployee names, employee contact information, employee work history,employee pay rate, and so forth. Depending on the needs of this retailcompany, the employee information and transactional information may bestored in different tables of the same database. The retail company mayhave a need to “query” its database when it wants to learn informationthat is stored in the database. This retail company may want to finddata about, for example, the names of all employees working at a certainstore, all employees working on a certain date, all transactions for acertain product made during a certain time frame, and so forth.

When the retail store wants to query its database to extract certainorganized information from the database, a query statement is executedagainst the database data. The query returns certain data according toone or more query predicates that indicate what information should bereturned by the query. The query extracts specific data from thedatabase and formats that data into a readable form. The query may bewritten in a language that is understood by the database, such asStructured Query Language (“SQL”), so the database systems can determinewhat data should be located and how it should be returned. The query mayrequest any pertinent information that is stored within the database. Ifthe appropriate data can be found to respond to the query, the databasehas the potential to reveal complex trends and activities. This powercan only be harnessed through the use of a successfully executed query.

Traditional database management requires companies to provisioninfrastructure and resources to manage the database in a data center.Management of a traditional database can be very costly and requiresoversight by multiple persons having a wide range of technical skillsets. Traditional relational database management systems (RDMS) requireextensive computing and storage resources and have limited scalability.Large sums of data may be stored across multiple computing devices. Aserver may manage the data such that it is accessible to customers withon-premises operations. For an entity that wishes to have an in-housedatabase server, the entity must expend significant resources on acapital investment in hardware and infrastructure for the database,along with significant physical space for storing the databaseinfrastructure. Further, the database may be highly susceptible to dataloss during a power outage or other disaster situations. Suchtraditional database systems have significant drawbacks that may bealleviated by a cloud-based database system.

A cloud database system may be deployed and delivered through a cloudplatform that allows organizations and end users to store, manage, andretrieve data from the cloud. Some cloud database systems include atraditional database architecture that is implemented through theinstallation of database software on top of a computing cloud. Thedatabase may be accessed through a Web browser or an applicationprogramming interface (API) for application and service integration.Some cloud database systems are operated by a vendor that directlymanages backend processes of database installation, deployment, andresource assignment tasks on behalf of a client. The client may havemultiple end users that access the database by way of a Web browserand/or API. Cloud databases may provide significant benefits to someclients by mitigating the risk of losing database data and allowing thedata to be accessed by multiple users across multiple geographicregions.

There exist multiple architectures for traditional database systems andcloud database systems. One example architecture is a shared-disksystem. In the shared-disk system, all data is stored on a sharedstorage device that is accessible from all processing nodes in a datacluster. In this type of system, all data changes are written to theshared storage device to ensure that all processing nodes in the datacluster access a consistent version of the data. As the number ofprocessing nodes increases in a shared-disk system, the shared storagedevice (and the communication links between the processing nodes and theshared storage device) becomes a bottleneck slowing data read and writeoperations. This bottleneck is further aggravated with the addition ofmore processing nodes. Thus, existing shared-disk systems have limitedscalability due to this bottleneck problem.

In some instances, it may be beneficial to access additional data thatis stored externally to a central or main database. In an example, adatabase user may have its data stored on its own internal systemsand/or cloud-based database account. The database user may wish toaccess its own data and may also wish to access additional data that isnot stored on the database user's internal systems and/or cloud-baseddatabase account. This additional data may be referred to as externaldata. In some instances, the database user may benefit from querying itsown data and the external data in tandem, or in comparing its own dataand the external data, or in modifying its own data according to changesmade to the external data, and so forth. The external data may be storedin one or more external tables. An external table represents contentthat is not directly managed by the database. There exist some databasestructures and file formats that are created to speed up query responsetimes and/or reduce the computing load to execute a database query. Onesuch database structure is a materialized view. Materialized viewsimprove performance of expensive database queries by materializing andreusing common intermediate query results in a workload. A materializedview is a database object that includes the results of a database queryand may improve performance of database queries by caching the resultsof the query. Materialized views can be costly to update and maintainsuch that efficient use of materialized views can be difficult toachieve.

Disclosed herein are systems, methods, and devices for maintaining,updating, and querying over external tables in database systems. Furtherdisclosed are systems, methods, and devices for generating andrefreshing materialized views over external tables in database systems.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive implementations of the presentdisclosure are described with reference to the following figures,wherein like reference numerals refer to like or similar partsthroughout the various views unless otherwise specified. Advantages ofthe present disclosure will become better understood with regard to thefollowing description and accompanying drawings where:

FIG. 1 is a schematic block diagram illustrating a system for generatinga materialized view on an external table, according to one embodiment;

FIG. 2 is a schematic block diagram illustrating a process flowinvolving an external data manager, according to one embodiment;

FIG. 3 is a schematic diagram of a data processing platform, accordingto one embodiment;

FIG. 4 is a block diagram of a resource manager, according to oneembodiment;

FIG. 5 is a block diagram of an execution platform, according to oneembodiment;

FIG. 6 is a block diagram of an external data manager, according to oneembodiment;

FIG. 7 is a block diagram of a process flow for querying over anexternal table, according to one embodiment;

FIG. 8 is a block diagram of a process flow for incremental updating ofa materialized view, according to one embodiment;

FIG. 9 is a block diagram of a process flow for updating a materializedview over an external table, according to one embodiment;

FIG. 10 is a schematic diagram of a data structure for storage of dataand metadata in a database system, according to one embodiment;

FIGS. 11A-11C are schematic diagrams of a data structure for versioningand refreshing of data and metadata in a database system, according toone embodiment;

FIG. 12 is a schematic flow chart diagram of a method for automaticallymaintaining an external table in a database system, according to oneembodiment;

FIG. 13 is a schematic flow chart diagram of a method for defining anexternal table in a database system, according to one embodiment;

FIG. 14 is a schematic flow chart diagram of a method for querying anexternal table in a database system, according to one embodiment;

FIG. 15 is a schematic flow chart diagram of a method for generating amaterialized view over an external table in a database system, accordingto one embodiment;

FIG. 16 is a schematic flow chart diagram of a method for querying anexternal table in a database system, according to one embodiment; and

FIG. 17 is a schematic diagram of an example computing device, accordingto one embodiment.

DETAILED DESCRIPTION

The systems, methods, and devices of the disclosure are directed toexternal tables in database systems. In an embodiment, a database systeminclude a multi-tenant and cloud-based database platform that enables aclient (i.e. an account associated with the database platform) to ingestdata into the database platform, store data in the database platform,and query the data in the database platform. In certain instances, aclient of the database platform may wish to query or analyze data thatis stored outside of the database platform and is not directly managedby the database platform. Such data may be referred to herein as“external data” or an “external table.”

In an embodiment, the database platform serves as a cloud datawarehouse. Data may be stored across shared storage devices of thedatabase platform and may be organized and managed by the databaseplatform. In an embodiment, structured data or semi-structured data maybe copied into a data bucket that is in communication with the databaseplatform. The data in the data buckets may be ingested by the databaseplatform and then organized and managed by the database platform. Theclient may then view and query such data.

However, in some instances, a client may not wish to ingest data intothe database platform or may wish to ingest only a portion of data intothe database platform. In such an instance, the client may benefitgreatly by the database platform reading and performing queries onexternal data. The external data is not managed directly by the databaseplatform, but the database platform may still query the external data,generate metadata about the external data, generate materialized viewsabout the external data, generate change tracking information about theexternal data, and so forth.

In an embodiment, a data lake is associated with the database platformand the client loads new data to the data lake. When data is loaded tothe data lake or when data in the data lake is modified or updated, thedatabase platform is notified. The database platform generates anexternal table based on the database platform and further generates andstores metadata about the data in the external table. The databaseplatform may generate materialized views based on the external tableand/or execute queries over the external table.

The above embodiment provides significant benefits for a client thatdoes not wish to have data ingested into the database platform but stillwishes to perform operations on the data by using the database platform.In an example, a client has adopted an external data storage platform asits data lake. In the example, the external data storage platform iscloud-based storage where data, including photos, videos, documents,tables, and other files, may be uploaded and stored. In an embodiment,the external data storage platform is a storage service that is referredto as a “data bucket” in some instances. In the example, the client hasadopted the external data storage platform as its data lake and wishesto consider the external data storage platform as its only source oftruth for its data. The client does not wish to copy any data into thedatabase platform. According to the systems, methods, and devicesdescribed herein, the client may still enjoy the benefits of using thedatabase platform without ingesting data into the database platform. Theclient may create an external table based on a selection of files on thedata bucket or based on all files in the data bucket. The external tablemay be provided to the database platform in a read-only manner such thatthe external table is not managed or manipulated by the databaseplatform. According to the embodiments disclosed herein, the client canuse the database platform to generate and update metadata about data inthe external table, query the external table, and generate materializedviews over the external table.

In a further example, a client wishes to use a mixture of external andinternal data, relative to the database platform. The client may wish tostore a first portion of its data in a data bucket with an external datastorage platform such that the data bucket serves as the client's datalake for the first portion of its data. The client may wish to store rawdata in the data bucket (serving as the client's data lake) such asimages, videos, non-partitioned tables, and other files. The client mayalso wish to store a second portion of its data with the databaseplatform such that the database platform serves as the client's datawarehouse. The data stored in the data warehouse (i.e., managed by thedatabase platform) is curated and organized to allow for highly granularquerying, sophisticated change tracking, and finely tuned analytics. Inthe example, the client may be motivated to have its data stored in twolocations for a number of reasons. One such reason is that the clientmay operate programs or applications that prefer to access data from thedata lake rather than a data warehouse, or in some instances may not becapable of accessing data from a data warehouse. Another reason may bethat the raw data stored in the data bucket cannot be processed by thedatabase platform. Another such reason may be that some types of data(e.g., applications logs of data-based events or actions in a system,network, or information technology environment) are generated frequentlyand rarely analyzed. The client may find it is inefficient to maintain adata ingestion pipeline with the database warehouse for application logsor other data that is unlikely to be queried frequently. Another reasonmay be that the client does not wish for certain data to be outside ofits virtual private cloud for security reasons. Another reason may bethat the client needs certain datasets to expire after a set amount oftime, and it may be challenging to purge those datasets from thedatabase warehouse. Another reason may be that the client wishes to usean open file format such as Apache Parquet such that the data isdirectly accessible to other applications without needing to access adata warehouse.

The above examples illustrate the benefits of the systems, methods, anddevices described herein for maintaining and querying over an externaltable. The described systems, methods, and devices enable a databaseplatform to generate metadata about external data that is stored outsidethe database platform and is not managed by the database platform.Additionally, the systems, methods, and devices described herein enablethe database platform to query the external data based on metadata thatis stored internally with the database platform. Additionally, thedescribed systems, methods, and devices described herein enable adatabase platform to generate materialized views over the external dataand refresh those materialized views in response to changes to theexternal data.

In an embodiment, a method for maintaining an external table in adatabase system is disclosed. The method includes receiving, by adatabase platform, read access to content stored in a data storageplatform that is separate from the database platform. The methodincludes defining an external table based on the content stored in thedata storage platform. The method includes connecting the databaseplatform to the external table such that the database platform has readaccess for the external table and does not have write access for theexternal table. The method includes generating metadata for the externaltable, the metadata comprising information about data stored in theexternal table. The method includes receiving a notification that amodification has been made to the content of the data storage platform,the modification comprising one or more of an insert, a delete, or anupdate. The method includes refreshing the metadata for the externaltable in response to the modification being made to the content of thedata storage platform.

In an embodiment, a method of querying over an external table isdisclosed. The method includes connecting a database platform to anexternal table such that the database platform has read access for theexternal table and does not have write access for the external table.The method includes receiving a query, the query directed at least todata in the external table. The method includes determining, based onmetadata, one or more files in the external table comprising datasatisfying the predicate. The method includes pruning, based on themetadata, all files in the external table that do not comprise any datasatisfying the query. The method includes generating a query plancomprising a plurality of discrete subtasks. The method includesassigning, based on the metadata, the plurality of discrete subtasks toone or more nodes in an execution platform.

In an embodiment, a method for generating a materialized view over anexternal table is disclosed. The method includes connecting a databaseplatform to an external table such that the database platform has readaccess for the external table and does not have write access for theexternal table. The method includes generating, by the databaseplatform, a materialized view over the external table. The methodincludes receiving a notification that a modification has been made tothe external table, the modification comprising one or more of new databeing added to the external table or existing data in the external tablebeing deleted or modified. The method includes in response to theexternal table being modified, refreshing the materialized view suchthat the materialized view comprises an accurate representation of theexternal table.

A client of a database platform may wish to store data in a datawarehouse and/or a data lake. A data lake is a storage repository thatcan hold a vast amount of raw data in its native format. A data lake mayserve as a single store of all data for a client, ranging from raw data(which may be an exact copy of source system data) to transformed data.The transformed data may be used for reporting, analysis, visualization,machine learning, and so forth. A data lake may be suited for storing alarge sum of data that may never make it into a data warehouse where itis managed, curated, and queried. Data lakes may store raw uncleanseddata and may be optimized for permitting the client to upload data inwhatever format is convenient for the client. In some implementations,data lakes present a file and file system abstraction that allows for alarge set of applications to operate on the data.

A data warehouse may store cleansed data in well-defined formats. Thedata in a data warehouse may be organized and optimized for analytics toenable highly granular querying and other processes. Data warehouses mayinclude relational schemas and more constrained processing modes sothere are fewer applications that can operate on data stored in a datawarehouse relative to data stored in a data lake.

In some instances, a data warehouse can be expensive to maintain. Theclient of the database platform may wish to store a bulk of its data ina data lake and only store curated and transformed data in a datawarehouse. Embodiments of the present disclosure enable the client tostore its data across a data lake and a data warehouse. Additionally,embodiments of the disclosure enable the client to store its data in adata lake and still enable the database platform to query and analyzethe data in the data lake.

In some instances, it may be desirable to generate one or morematerialized views over an external table. This may be particularlybeneficial where a client wishes to query data in the external tablethat is stored externally to a database platform and is not managed bythe database platform. Because the external data is not managed by thedatabase platform, it may be particularly difficult or inefficient toaccess and/or read the external data. A materialized view is a databaseobject that includes the results of a query. A materialized view may bea local copy of data that is located remotely or may be a subset of therows and/or columns of a table or join result or may include a summaryusing an aggregate function. The process of generating a materializedview may be referred to as materialization. Materialization is a form ofcaching the results of a query to a concrete “materialized” table thatcan be updated based on changes made to the original base table (in thiscase, the base table is the external table). The materialized viewenables efficient access and decreases the amount of time and computingresources that are required to respond to a query. If the same queryparameters are requested again, the materialized view may be referencedrather than the base table (i.e. the external table). The materializedview is managed by the database platform and can be read more quicklyand efficiently than the external storage where the external table islocated.

In an example, a client wishes to maintain all data in a data lake thatis external to a database platform. The client wishes to use thedatabase platform to analyze and query the data that is stored in thedata lake. The database platform does not manage the data lake.According to the systems, methods, and devices disclosed herein, theclient may create an external table that is stored in the data lake andis accessible to the database platform. The database platform maygenerate one or more materialized views over the external table suchthat the one or more materialized views are stored, managed, andrefreshed by the database platform. When the client sends a queryrequest to the database platform, the database platform may respond tothe query by reading the external table directly from the data lake.However, to reduce the amount of time and processing resources that arerequired to respond to the query, the database platform may insteadrespond to the query by referencing the one or more materialized viewsthat are generated over the external table.

In an embodiment, and further to the above example, the databaseplatform responds to the query by referencing metadata about theexternal table. The metadata is generated, maintained, stored, andrefreshed by the database platform. The metadata is information aboutthe data within the external table. The metadata includes informationabout how the data in the external table is stored. In an embodiment,the data in the external table (stored in the external storage platform)may be organized as a source directory hierarchy where the leaves of thehierarchy are files. Each source directory can include one or more filesand the source directory can be modeled as a partition on the externaltable. The external table may include millions of rows of data and maybe very large and difficult to store or read. In an example, theexternal table is divided into six distinct partitions, and each of thesix partitions includes a portion of the data in the external table.Dividing the external table data into multiple partitions helps toorganize the data and to find where certain data is located within thetable.

An analogy to the partitions of the external table may be differentstorage buildings within a storage compound. In the analogy, the storagecompound is similar to the external table, and each separate storagebuilding is similar to a partition. Hundreds of thousands of items arestored throughout the storage compound. Because so many items arelocated at the storage compound, it is necessary to organize the itemsacross the multiple separate storage buildings. The items may beorganized across the multiple separate storage buildings by any meansthat makes sense. For example, one storage building may store clothing,another storage building may store household goods, another storagebuilding may store toys, and so forth. Each storage building may belabeled so that the items are easier to find. For example, if a personwants to find a stuffed bear, the person will know to go to the storagebuilding that stores toys. The storage building that stores toys mayfurther be organized into rows of shelving. The toy storage building maybe organized so that all stuffed animals are located on one row ofshelving. Therefore, the person looking for the stuffed bear may know tovisit the building that stores toys and may know to visit the row thatstores stuffed animals. Further to the analogy with database technology,each row of shelving in the storage building of the storage compound maybe similar to a column of database data within a partition of theexternal table. The labels for each storage building and for each row ofshelving are similar to metadata in a database context.

Similar to the analogy of the storage compound, the partitions disclosedherein can provide considerable benefits for managing data, findingdata, and organizing data. Each partition organizes data into rows andcolumns and stores a portion of the data associated with an externaltable (or a database table that is stored and managed by the databaseplatform). One external table or internal table may have manypartitions. The partitioning of the database data among the manypartitions may be done in any manner that makes sense for that type ofdata. For example, if the client is a credit card provider and the datais credit card transactions, the table may include columns such ascredit card number, account member name, merchant name, date of cardtransaction, time of card transaction, type of goods or servicespurchased with card, and so forth. The table may be stored within a datalake or data warehouse that is external to the database provider and isnot managed or updated by the database provider. The table may includemillions of credit card transactions spanning a significant time period,and each credit card transaction may be stored in one row of the table.Because the table includes many millions of rows, the table may bepartitioned into partitions. In the case of credit card transactions, itmay be beneficial to split the table based on time. For example, eachpartition may represent one day or one week of credit card transactions.It should be appreciated that the table may be partitioned intopartitions by any means that makes sense for the database client and forthe type of data stored in the table. The partitions provide significantbenefits for managing the storage of the millions of rows of data in thetable, and for finding certain information in the table.

A query may be executed on an external or internal table to find certaininformation within the table. Further to the above example of the creditcard transactions, a query may seek to find all transactions for acertain vendor across a certain time period. For example, a client (inthis example, the credit card provider) may send a query to the databaseprovider and ask for a report of all credit transactions that occurredat Retail Store A in the months of January, April, and May. In theexample, these credit card transactions are stored in an external tablethat is not managed or updated by the database platform. To respond tothe query, a resource manager (see 302) of the database platform mustscan the external table to find each of the applicable credit cardtransactions. The external table may include millions and millions ofrows, and it would be very time consuming and it would requiresignificant computing resources for the resource manager to scan theentire external table. The partition organization along with thesystems, methods, and devices for external table metadata storage asdisclosed herein provide significant benefits by shortening the queryresponse time and reducing the amount of computing resources that arerequired for responding to the query.

Further to the above example, the resource manager must respond to thequery that requested all credit card transactions at Retail Store A inthe months of January, April, and May. The credit card transactions arestored in an external table that is not managed or updated by thedatabase platform or the resource manager. The resource manager may findthe cells of data by scanning external table metadata. The externaltable metadata includes information about the data stored within theexternal table, but the external table metadata is generated, stored,and refreshed by the database platform and is considered “internal” tothe database platform. The multiple level database metadata as describedherein (see e.g. FIG. 10) enables the resource manager to quickly andefficiently find the correct data to respond to the query. The resourcemanager may find the correct table by scanning cumulative table metadata(see 1002) across all the multiple tables in the client's database. Theresource manager may find a correct grouping of partitions by scanningmultiple grouping expression properties (see 1014 a-1014 d) across theidentified table. The grouping expression properties include informationabout data stored in each of the partitions within the grouping. Theresource manager may find a correct partition by scanning multiplepartition expression properties within the identified grouping ofpartitions. The resource manager may find a correct column by scanningone or more column expression properties within the identifiedpartition. The resource manager may find the correct row(s) by scanningthe identified column within the identified partition. Further to theexample involving the credit card transactions, the resource manager mayscan multiple cumulative table metadata to find a table that includescredit card transactions. The resource manager may scan the groupingexpression properties to find groupings that have data for the months ofJanuary, April, and/or May. For example, the resource manager finds agrouping that includes credit card transactions for the month of January(and may further include transactions for other months). The resourcemanager reads the partition expression properties for that grouping tofind one or more individual partitions that include transactions for themonth of January. The resource manager reads column expressionproperties within each of the identified individual partitions. Theresource manager scans the identified columns to find the applicablerows that have a credit card transaction for Retail Store A in the monthof January (or April or May). The metadata points the resource managerto the correct external table and to the correct partition of theexternal table. The metadata may further point the resource manager tothe correct rows of that partition of the external table. By scanningthe metadata that is stored internally to the database platform, theresource manager may quickly find where the correct data is storedwithin the external table.

As illustrated in the above example, the metadata organization asdisclosed herein provides significant benefits to hasten the queryresponse time and enables the resource manager to quickly identify thecorrect external table, the correct partition within the external table,and the correct columns and rows within the external table to respond toa query. The novel metadata storage as disclosed herein provides amultiple level metadata structure for maintaining information aboutpartitions in an external table.

Before the methods, systems, and devices for a database platform toperform operations over an external table are disclosed and described,it is to be understood that this disclosure is not limited to theconfigurations, process steps, and materials disclosed herein as suchconfigurations, process steps, and materials may vary somewhat. It isalso to be understood that the terminology employed herein is used fordescribing implementations only and is not intended to be limiting sincethe scope of the disclosure will be limited only by the appended claimsand equivalents thereof.

To provide further background to the disclosures provided herein, atable (including an external table or an internal database table) is acollection of related data held in a structured format and may includecolumns and rows. A table may be altered in response to a datamanipulation (DML) command such as an insert command, a delete command,an update command, a merge command, and so forth. Such modifications maybe referred to as a transaction that occurred on the table. In anembodiment, each transaction includes a timestamp indicating when thetransaction was received and/or when the transaction was fully executed.In an embodiment, a transaction includes multiple alterations made to atable, and such alterations may impact one or more partitions in thetable. In an embodiment, data may be continuously ingested, or may beingested at determined time intervals, and the ingestion of data intothe table is a transaction occurring on the table. In an embodiment,each time a transaction is executed on the table, a new table version isgenerated that includes one or more new partitions. Further, each time atransaction is executed on the table, or after a threshold number oftransactions are executed on the table, the metadata for the table mayneed to be updated to reflect the new or updated data stored in thetable.

A table may store data in a plurality of partitions, wherein thepartitions are immutable storage devices. When a transaction is executedon a such a table, all impacted partitions are recreated to generate newpartitions that reflect the modifications of the transaction. After atransaction is fully executed, any original partitions that wererecreated may then be removed from the table. A new version of the tableis generated after each transaction that is executed on the table. Thetable may undergo many versions over a time period if the data in thetable undergoes many changes, such as inserts, deletes, updates, and/ormerges. Each version of the table may include metadata indicating whattransaction generated the table, when the transaction was ordered, whenthe transaction was fully executed, and how the transaction altered oneor more rows in the table. The disclosed systems, methods, and devicesfor low-cost table versioning may be leveraged to provide an efficientmeans for updating table metadata after one or more changes(transactions) have occurred on the table. Such systems, methods, anddevices may be implemented for generating metadata that is storedinternally with a database platform, wherein the metadata includesinformation about data stored in an external table that is not managedor updated by the database platform.

In one embodiment, metadata is stored and maintained on non-mutablestorage services (may be referred to herein as micro-partitions) in thecloud. These storage services may include, for example, Amazon S3 ®,Microsoft Azure Blob Storage®, and Google Cloud Storage®. Many of theseservices do not allow to update data in-place (i.e., are non-mutable orimmutable). Data micro-partitions may only be added or deleted, butnever updated. In one embodiment, storing and maintaining metadata onthese services requires that, for every change in metadata, a metadatapartition is added to the storage service. These metadatamicro-partitions may be periodically consolidated into larger“compacted” or consolidated metadata micro-partitions in the background.

In an embodiment, all data in tables is automatically divided into animmutable storage device referred to as a micro-partition. Themicro-partition may be considered a batch unit where each partition hascontiguous units of storage. By way of example, each micro-partition maycontain between 50 MB and 500 MB of uncompressed data (note that theactual size in storage may be smaller because data may be storedcompressed). Groups of rows in tables may be mapped into individualpartitions organized in a columnar fashion. This size and structureallow for extremely granular selection of the partitions to be scanned,which can be comprised of millions, or even hundreds of millions, ofmicro-partitions. This granular selection process may be referred toherein as “pruning” based on metadata. Pruning involves using metadatato determine which portions of a table, including which micro-partitionsor micro-partition groupings in the table, are not pertinent to a query,and then avoiding those non-pertinent micro-partitions when respondingto the query and scanning only the pertinent micro-partitions to respondto the query. Metadata may be automatically gathered about all rowsstored in a micro-partition, including: the range of values for each ofthe columns in the micro-partition; the number of distinct values;and/or additional properties used for both optimization and efficientquery processing. In one embodiment, partitioning may be automaticallyperformed on all tables. For example, tables may be transparentlypartitioned using the ordering that occurs when the data isinserted/loaded.

In describing and claiming the disclosure, the following terminologywill be used in accordance with the definitions set out below.

It must be noted that, as used in this specification and the appendedclaims, the singular forms “a,” “an,” and “the” include plural referentsunless the context clearly dictates otherwise.

As used herein, the terms “comprising,” “including,” “containing,”“characterized by,” and grammatical equivalents thereof are inclusive oropen-ended terms that do not exclude additional, unrecited elements ormethod steps.

As used herein, a database table is a collection of records (rows). Eachrecord contains a collection of values of table attributes (columns).Database tables are typically physically stored in multiple smaller(varying size or fixed size) storage units, e.g. files or blocks.

A detailed description of systems and methods consistent withembodiments of the present disclosure is provided below. While severalembodiments are described, it should be understood that this disclosureis not limited to any one embodiment, but instead encompasses numerousalternatives, modifications, and equivalents. In addition, whilenumerous specific details are set forth in the following description inorder to provide a thorough understanding of the embodiments disclosedherein, some embodiments may be practiced without some or all of thesedetails. Moreover, for the purpose of clarity, certain technicalmaterial that is known in the related art has not been described indetail in order to avoid unnecessarily obscuring the disclosure.

Referring now to the figures, FIG. 1 illustrates a schematic blockdiagram of a system 100 for generating, refreshing, and querying anexternal table. The system 100 includes a database platform 102. Thedatabase platform 102 may be a cloud-based database computing platformfor storing, organizing, maintaining, and querying database data. Thedatabase platform 102 may be in communication with multiple clientaccounts. An embodiment of the database platform 102 includes the system300 illustrated in FIG. 3. The database platform 102 is in communicationwith a data lake 104 that receives data from a client 118. The data lake104 may include a cloud-based data storage resource that may receivefiles and raw data in its native format.

An external table 106 may be generated based on data within the datalake 104. The external table 106 includes structured or semi-structureddata. The external table 106 is accessible to the database platform 102but is not managed or updated by the database platform 102. In anembodiment, the external table 106 is stored within the data lake 104and the structure and organization of the external table 106 is definedby the client 118. In an embodiment, the structure and organization ofthe external table 106 is defined by the database platform 102. The datalake 104 includes a store of data that is managed by the client 118,wherein the client 118 is associated with the data lake 104 and thedatabase platform 102. The data lake 104 may be external to the databaseplatform 102 such that the database platform 102 does not have theability or authorization to write or manipulate the data within the datalake 104. The database platform 102 may have permissions to read thedata stored in the data lake 104, query the data stored in the data lake104, and/or receive an indication when the data lake 104 is updated.

In an embodiment, the external table 106 is generated based on a sourcedirectory in the data lake 104. The source directory in the data lake104 may alternatively be referred to as a namespace or source file. Thesource directory may be identified by the client 118 and the client maymanually indicate that new data should be uploaded to the sourcedirectory. The data lake 104 is a system or repository of data. Datawithin the data lake 104 may be stored in a structured or unstructuredstate at any scale. The data may be stored as-is without firststructuring the data. In an embodiment, the data lake 104 is a singlestore of all enterprise data for the client 118, including raw copies ofsource system data and transformed data that may be utilized for taskssuch as reporting, visualization, analytics, machine learning, and soforth. In an embodiment, the data lake 104 includes structured data fromrelational databases (i.e. rows and columns), semi-structured data,unstructured data (e.g. emails, documents, and so forth), and binarydata (e.g. images, audio, video, and so forth). The data lake 104 maymanage big data for the client 118 by providing a single point ofcollecting, organizing, and sharing data.

The data lake 104 may be distinguished from a data warehouse. However,in certain embodiments, a data warehouse may be utilized rather than adata lake 104 is illustrated in FIG. 1. A data warehouse includes adatabase optimized to analyze relational data coming from transactionsystems and line of business applications. In a data warehouse, the datastructure and schema may be defined in advance to optimize for fastqueries. The data in a data warehouse may be cleaned, enriched, andtransformed such that it acts as a single source of truth. The data lake104 may store relational data from line of business applications andnon-relational data from, for example, mobile applications, internet ofthings devices, and social media. In a data lake 104, the structure orschema of the data may not be defined when data is captured such thatall data may be stored without careful design. In certainimplementations, it may be beneficial to employ both a data warehouseand a data lake, and such an implementation may benefit from thesystems, methods, and devices for generating a materialized view basedon an external table, as disclosed herein.

The data lake 104 may import any amount of new data 108 that may beingested in real time. The new data 108 may be collected from multiplesources and moved to the data lake 104 in its original format. Thisprocess may permit the data lake 104 to scale data of any size whilesaving time defining data structures, schema, and transformations. Thedata lake 104 may provide an ability to understand what data is storedin the data lake 104 through crawling, cataloging, and indexing thedata. The data within the data lake 104 may be secured and encrypted toensure the data is protected.

The data lake 104 provides directories for the external table 106 andmaterialized view 116. The directories may be stored in a different fileformat than the tables stored in the shared storage devices of thedatabase platform 102. The directories of the data lake 104 may bestored in a cloud storage system such as Amazon Web Services™, MicrosoftAzure™, and so forth. The data lake 104 may be separate and independentof the database platform 102 (see e.g. FIGS. 3-5).

The client 118 may add new data 108 to the data lake 104. The new data108 may be in any file format In an embodiment, the new data 108 must bein a specific file format to be read by the database platform 102 or tobe incorporated into the external table 106. In an embodiment, thedatabase platform 102 includes shared storage devices for storingdatabase data (this may be referred to as “internal” data that ismanaged by the database platform 102). The client 118 may have datastored in the data lake 104 and may further have different or replicateddata that is stored in the shared storage devices of the databaseplatform 102. In an embodiment, the data stored in the data lake 104 andthe data stored in the shared storage devices of the database platform102 have different file formats. In such an embodiment, the databaseplatform 102 may be configured to read the different formats for datastored in the data lake 104 and/or convert those different formats tothe same data format used by the database platform 102.

When the new data 108 is added to the data lake 104, a notification 110is generated and provided to an ingest service 112 of the databaseplatform 102. The notification 110 includes an indication that the newdata 108 has been added to the data lake 104. The notification 110 maybe automatically generated by the data lake 104, may be automatically ormanually generated by the client, may be automatically or manuallyretrieved by the database platform 102, and so forth. In an embodiment,the database platform 102 queries the data lake 104 at threshold periodsto determine whether new data 108 has been added to the data lake 104.In an embodiment, the notification 110 is provided to the ingest service112 to indicate that new data 108 has been received by the data lake104. The ingest service 112 may prompt an update to be made to theexternal table 106 and further to a materialized view 116 based on thenew data 108.

The ingest service 112 receives notifications from the data lake 104that an update has been made do data stored within the data lake 104.The ingest service 112 may receive a notification that data within thesource directory (may alternatively be referred to as a namespace orsource file) in the data lake has been updated. The ingest serve 112updates metadata for the external table 106 to reflect any modificationsmade to the data lake 104 data.

The new data 108 is read and/or retrieved by the ingest service 112 ofthe database platform 102. The new data 108 is stored in an externaltable 106. The external table may be in communication with the databaseplatform 102 but may not be managed by the database platform. In anembodiment, the database platform 102 can read data in the externaltable 106 but cannot write data to the external table 106. In anembodiment, the client 118 manages the external table 106 and providesaccess to the external table 106 to the database platform 102. In anembodiment, the external table 106 is managed and/or provided by acloud-based data warehousing service that may be separate andindependent of the database platform 102. When new data 108 is added tothe data lake 104, the external table 106 is updated to reflect the newdata 108.

The database platform 102 may generate one or more materialized views116 based on the external table 106. A materialized view 116 over theexternal table 106 may be managed by the database platform 102 and maybe stored in the plurality of shared storage devices (see e.g. 308) thatare associated with the database platform 102. When new data 108 isadded to the data lake 104, the external table 106 is updated and thematerialized view 116 over the external table 106 is refreshed.

The database platform 102 generates the materialized view 116 over theexternal table 106. The generation of the materialized view 116 over theexternal table 106 may be decomposed into two steps. In a first step, anon-materialized external table is generated. In a second step, amaterialized view is generated over the non-materialized external table.In certain implementations, it may be beneficial to decompose thenon-materialized external table from the materialized view as disclosedherein. Decomposing may allow for multiple different materialized viewsto be generated, and each of the multiple materialized views may includea different selection of data, different projections, differentsummaries, and so forth, without first materializing all of the sourcedata.

The combination of a non-materialized external table 106 and amaterialized view 116 over the external table 106 may be beneficial inimplementations where a large amount of data is already stored in a datalake 104 and only the most recent subset of that data is frequentlyqueried. In such an implementation, it may be cost-prohibitive tomaterialize all of the data in the data lake 104. Further, generating amaterialized view over an external table may permit materialization ofonly the portion of the data in the data lake 104 that is queried mostfrequently.

The metadata component 114 generates and refreshes metadata based on thedata in the data lake 104 and/or the external table 106. The metadata isgenerated, managed, stored, and refreshed by the database platform 102.The metadata is information about the data stored in the data lake 104and/or the external table 106. The metadata may be organized accordingto the improved metadata systems disclosed herein, for example thosedepicted in FIG. 10 and FIGS. 11A-11C. The metadata includes informationabout the data stored in the data lake 104 and/or the external table 106such that a resource manager (see e.g. 302) of the database platform 102can execute queries over the external table 106 without reading all datain the external table.

The metadata component 114 further generates and refreshes metadataabout the materialized view 116. When the data lake 104 and/or theexternal table 106 are updated, the materialized view 116 may berefreshed to reflect the updates. The metadata may further be refreshedby the metadata component 114 to reflect the updates made to thematerialized view 116. When the client 118 queries the external table106, the resource manager (see 302) of the database platform 102 mayexecute the query on the materialized view 116 if a materialized view116 exists that can respond to the query. The database platform 102expends less time and processing resources when the query is executedover the materialized view 116 rather than the external table 106itself.

In an embodiment, the data in the external table 106 is organized intopartitions that constitute immutable storage devices. In an embodiment,the data in the external table 106 is organized into mutable storagedevices that can be updated in-place, but the database platform 102interacts with the external table 106 as if the data cannot be updatedin-place. The data in the external table 106 may be stored in a fileformat that is different from those file formats commonly used forinternal data associated with the database platform. The resourcemanager 302 of the database platform 102 can scan the metadata todetermine which partitions in the external table 106 need to be consumedto respond to a query.

In an embodiment, the client 118 issues a command to the databaseplatform to create the external table 106 and the command does notdefine a schema for the external table 106. In such an embodiment, theexternal table 106 is generated with no schema defined with the tabledefinition. The external table 106 may include a variant column and alldata in the external table 106 may be queried using the smart column.

The client 118 may query external table 106 metadata by way of variousmethods. In an embodiment, a view shows all external tables in thedatabase. The columns in such a result are similar to those of regulartables, while the external table 106 will have additional columns. In anembodiment, the external table 106 includes a notification channelcolumn that specifies a resource name of a simple queue service (SQS)queue that is created in the back end such that a client account maysetup automatic addition of files to the external table 106. In anembodiment, the external table 106 includes a location column thatspecifies the location which the external table 106 is configured with.

A metadata view indicates the directories that supply data to theexternal table 106. Where the source of the data is the data lake 104,new files may be added at any time, old files may be deleted at anytime, and files may be updated at any time. In such an embodiment, itmay be beneficial to generate a metadata view of all files that supplydata to the external table 106. In an embodiment, all directories forthe external table 106 are tracked and such data is available to view inan information schema. directories that are de-registered or deleted maybe removed from the metadata view.

The materialized view 116 may be generated over an external table 106 toprovide for faster query response time. The materialized view 116 may beautomatically and incrementally updated to ensure that data is alwaysup-to-date with a primary source of truth, such as a source directory inthe data lake 104. The fast query 116 may be processed against thematerialized view 116 to improve query response time on frequently useddata.

FIG. 2 is a schematic block diagram of a process flow 200 for managingexternal tables in a database platform 102. The system 200 includes andatabase platform 102 in communication with a data lake 204 and a client218. The database platform 102 may be incorporated into a resourcemanager (see e.g. 302) of a database platform 102 as disclosed herein.

The client 218 is in communication with a data lake 204 that is managedby or connected with the client 218 and is external to the databaseplatform 102. In an embodiment, the database platform 102 is incommunication with the data lake 204 such that the database platform 102can read data written to the data lake 204 but cannot write data to thedata lake 204. The database platform 102 may be incorporated into acloud-based database computing platform and the data lake 204 mayconstitute a separate and independent cloud-based storage structure.

In the process flow 200, the client 218 adds data to the data lake 204.A notification listener component 220 receives an indication that datawas added to the data lake 204. The notification may be generatedautomatically or manually by the data lake 204 or the client 218, Thenotification listener component 220 provides an indication to anexternal table refresh component 222 that data was added to the datalake 204 and an external table should be refreshed or updated to reflectthe new data that was added to the data lake 204. The notificationlistener component 220 may be in communication with a source table inthe data lake 204 that serves as a source table for an external tablethat may be queried by the database platform 102. The client 218 maydefine the source table in the data lake 208 and the notificationlistener component 220 may receive a notification whenever data is addedto or updated in the source table.

The external table refresh component 222 refreshes an external tablethat is readable to the database platform 102. In an embodiment, thedatabase platform 102 is a component of a cloud-based database computingplatform that organizes database data (that is managed and stored by thecloud-based database computing platform) into immutable storage devicesreferred to herein as partitions. In such an embodiment, the data addedto the data lake 204 may not be organized into partitions. The externaltable refresh component 222 may specify one or more partitions for theexternal table based on the new data that was added to the data lake204. In an embodiment, the data in the data lake 204 is of a differentfile format compared with internal data that is managed by thecloud-based database platform. The external table refresh component 222may be configured to read and understand multiple file formats andtranslate data into the same file format that is used by the cloud-baseddatabase platform.

The metadata component 224 generates, organizes, and stores metadataabout the data stored in the external table. The metadata is based onthe partitions for the external data that are specified by the externaltable refresh component 222. Example structures for the metadata can beseen in FIG. 10 and FIGS. 11A-11C. In an embodiment, the client 218specifies a hierarchical structure for the data within the externaltable. The client 218 may specify how folders and subfolders will beorganized for the external data, how the data will be partitioned intorows and columns, and/or how the external data will be partitioned intopartitions. In an example, the client 218 may store insuranceinformation in the external table. In the example, the client 218 mayspecify that the data should be organized with separate columns for theyear, month, date, and time of an insurance transaction. The client 218may specify that the insurance transactions should be organized bylocation such that each state or region is stored in a differentpartition. In an embodiment, the client 218 uploads a file to a specificfolder in the data lake 204, and the location of the file guides themetadata component 224 to generate the correct metadata for the file.

In an embodiment, the metadata component 224 generates change trackingmetadata for a partition in the external table that has been modified.The notification listener component 220 may receive an indication that acertain file in the data lake 204 was modified. The certain subfolderwithin the source directory may provide data for a certain partition inthe external table. Based on the notification, database platform mayupdate the metadata about the external table.

The materialized view refresh component 226 updates a materialized viewthat is generated over the external table. The materialized view may bestored and managed by the cloud-based database computing platform thatis separate from the data lake 204. The materialized view is refreshedto reflect any new data added to the data lake 204 and/or to reflect anyupdates made to the external data, such as merges or deletes.

The external table generation component 228 generates an external tablebased on data within the data lake 204. The external table may bedefined by the external table generation component 228 or the client218. In an embodiment, the client 218 provides the database platform 102with access to a source directory in the data lake 204 and indicatesthat an external table should be generated based on the sourcedirectory.

In an embodiment, the client 218 provides an indication of ahierarchical structure in the source directory, wherein the hierarchicalstructure may have been manually defined by the client 218. Thehierarchical structure may indicate organization for the data in thesource directory, including folders and subfolders for the data. In anexample, the client 218 may indicate that one folder includes all creditcard transactions for the year 2019. The client 218 may further indicatethat a first subfolder includes all credit card transactions for Januaryin the year 2019, a second subfolder includes all credit cardtransactions for February in the year 2019, and so forth. The client mayfurther provide an indication of a partitioning structure for theexternal table that will be based on the source directory. Thepartitioning structure may indicate how data in the external tableshould be organized into columns, rows, and partitions. The externaltable generation component 228 may define the external table based onthe hierarchical structure, the partitioning structure, and where thefiles are uploaded within the folders and subfolders of the hierarchicalstructure.

In an example, the client 218 indicates that the partitions in theexternal table should be organized based on location such that allcredit card transactions in the state of California are in onepartition, all credit card transactions in the state of Maine are in adifferent partition, and so forth. The client 218 may further indicatethat the partitions in the external table should further be organizedbased on the timestamp for the credit card transaction. The client mayupload credit card transactions for the state of California in the monthof April in the year of 2019 in a folder for California, in a subfolderfor the year of 2019 in California, and further in a subfolder for themonth of April in the year of 2019 in California. Based on thehierarchical structure and the partitioning structure, the externaltable generation component 228 defines a partition in the external tablethat includes data for credit card transactions in California in Aprilof 2019.

Folders and subfolders may be stacked within the source directory of thedata lake and may be defined by the client. When data is uploaded to acertain subfolder in the source directory, the external table generationcomponent 228 defines the external table based on where the data wasuploaded in the context of the hierarchical structure and thepartitioning structure that were defined by the client.

The query component 230 receives a query from the client 218. The querycomponent 230 scans the metadata to determine one or more partitionsthat are necessary to respond to the query. In an embodiment, at leastone of the one or more partitions is stored in an external table that isnot managed or stored by the cloud-based database computing platform.The query component 230 parses execution of the query into a pluralityof discrete subtasks and assigns the discrete subtasks to multiple nodesof an execution platform. The execution platform executes the query.

The security component 232 manages permissions for the external table.When the database platform 102 has permission to view data in theexternal table, the query component 218 can access that data to respondto the query.

Referring now to FIG. 3, a data processing platform 300 is illustratedfor running the methods and systems disclosed herein. As shown in FIG.3, resource manager 302 may be coupled to multiple client accounts 314a, 314 b, and 314 n. The client accounts 314 a, 314 b, and 314 n mayrepresent different clients such as the client 118 illustrated inFIG. 1. In particular implementations, the resource manager 302 cansupport any number of client accounts desiring access to the executionplatform 304 and/or shared database storage 308. Client accounts 314 a,314 b, and 314 n may include, for example, end users providing userfiles to be ingested into the database, data storage and retrievalrequests, system administrators managing the systems and methodsdescribed herein, and other components/devices that interact withresource manager 302.

The resource manager 302 provides various services and functions thatsupport the operation of all systems and components within the dataprocessing platform 300. The resource manager 302 may be coupled toshared metadata 312, which is associated with the entirety of datastored throughout data processing platform 300. The shared metadata 312includes metadata for data stored in the shared database storage 308 andfurther includes metadata for data stored in external tables (see 106).In some embodiments, shared metadata 312 includes a summary of datastored in remote data storage systems as well as data available from alocal cache. Additionally, shared metadata 312 may include informationregarding how data is organized in the remote data storage systems andthe local caches. Shared metadata 312 may allow systems and services todetermine whether a piece of data needs to be processed without loadingor accessing the actual data from a storage device. The shared metadata312 may be organized according to the metadata structures illustrated inFIG. 10 and FIGS. 11A-11C.

The resource manager 302 may be further coupled to the executionplatform 304, which provides multiple computing resources that executevarious data storage and data retrieval tasks, as discussed in greaterdetail below. The execution platform 304 includes a plurality ofexecution nodes 306 a, 306 b, 306 c, and 306 n configured to processvarious tasks associated with the database, including ingesting new userfiles and generating one or more partitions for a table (may be anexternal table or a table stored in the shared database storage 308)based on the new user files. The execution platform 304 may be coupledto the shared database storage 308 including multiple data storagedevices 310 a, 310 b, 310 c, and 310 n. In some embodiments, the shareddatabase storage 308 includes cloud-based storage devices located in oneor more geographic locations. For example, the shared database storage308 may be part of a public cloud infrastructure or a private cloudinfrastructure. The shared database storage 308 may include hard diskdrives (HDDs), solid state drives (SSDs), storage clusters or any otherdata storage technology. Additionally, shared database storage 308 mayinclude distributed file systems (such as Hadoop Distributed FileSystems (HDFS)), object storage systems, and the like. It should beappreciated that the shared database storage 308 may be accessible byone or more instances of the resource manager 302 but may not beaccessible by all client accounts 314 a-314 n. In an embodiment, asingle instance of the resource manager 302 is shared by a plurality ofclient accounts 314 a-314 n. In an embodiment, each client account 314a-314 n has its own resource manager and/or its own shared databasestorage 308 that is shared amongst a plurality of execution nodes 306a-306 n of the execution platform 304. In an embodiment, the resourcemanager 302 is responsible for providing a particular client account 314a-314 n access to particular data within the shared database storage308.

In particular embodiments, the communication links between the resourcemanager 302 and client accounts 314 a-314 n, shared metadata 312, andexecution platform 304 are implemented via one or more datacommunication networks. Similarly, the communication links betweenexecution platform 304 and shared database storage 308 are implementedvia one or more data communication networks. These data communicationnetworks may utilize any communication protocol and any type ofcommunication medium. In some embodiments, the data communicationnetworks are a combination of two or more data communication networks(or sub-networks) coupled to one another. In alternative embodiments,these communication links are implemented using any type ofcommunication medium and any communication protocol.

As shown in FIG. 3, data storage devices 310 a-310 n are decoupled fromthe computing resources associated with execution platform 304. Thisarchitecture supports dynamic changes to data processing platform 300based on the changing data storage/retrieval needs as well as thechanging needs of the users and systems accessing data processingplatform 300. This architecture enables the execution platform 304 andthe shared database storage 308 to be effectively infinitely scalable.The support of dynamic changes allows the data processing platform 300to scale quickly in response to changing demands on the systems andcomponents within data processing platform 300. The decoupling of thecomputing resources from the data storage devices supports the storageof large amounts of data without requiring a corresponding large amountof computing resources. Similarly, this decoupling of resources supportsa significant increase in the computing resources utilized at aparticular time without requiring a corresponding increase in theavailable data storage resources.

The resource manager 302, shared metadata 312, execution platform 304,and shared database storage 308 are shown in FIG. 3 as individualcomponents. However, each of the resource manager 302, the sharedmetadata 312, the execution platform 304, and the shared databasestorage 308 may be implemented as a distributed system (e.g.,distributed across multiple systems/platforms at multiple geographiclocations). Additionally, each of resource manager 302, shared metadata312, execution platform 304, and shared database storage 308 can bescaled up or down (independently of one another) depending on changes tothe requests received from client accounts 314 a-314 n and the changingneeds of data processing platform 300. Thus, data processing platform300 is dynamic and supports regular changes to meet the current dataprocessing needs.

FIG. 4 is a block diagram depicting an embodiment of resource manager302. As shown in FIG. 4, resource manager 302 includes an access manager402 and a key manager 404 coupled to a data storage device 406. Accessmanager 402 may handle authentication and authorization tasks for thesystems described herein. Key manager 404 may manage storage andauthentication of keys used during authentication and authorizationtasks. A request processing service 408 manages received data storagerequests and data retrieval requests. A management console service 410supports access to various systems and processes by administrators andother system managers.

The resource manager 302 may also include a job compiler 412, a joboptimizer 414 and a job executor 416. Job compiler 412 parses tasks,such as ingest tasks, and generates the execution code for the ingestionof user files. Job optimizer 414 determines the best method to executeingest tasks based on the data that needs to be processed and/oringested. Job executor 416 executes code for ingest tasks received byresource manager 302. A job scheduler and coordinator 418 may sendreceived user files to the appropriate services or systems forcompilation, optimization, and dispatch to the execution platform 304. Avirtual warehouse manager 420 manages the operation of multiple virtualwarehouses implemented in an execution platform.

Additionally, the resource manager 302 includes a configuration andmetadata manager 422, which manages the information related to the datastored in the remote data storage devices and in the local caches. Amonitor and workload analyzer 424 oversees the processes performed byresource manager 302 and manages the distribution of tasks (e.g.,workload) across the virtual warehouses and execution nodes in theexecution platform. Configuration and metadata manager 422 and monitorand workload analyzer 424 are coupled to a data storage device 426.

The resource manager 302 includes an database platform 102 as describedin FIG. 2. The database platform 102 manages the interaction between thedatabase platform 102 and an external table 106. The external table mayinclude data stored in a source table of a data lake 104.

FIG. 5 is a block diagram depicting an embodiment of an executionplatform 304. As shown in FIG. 5, execution platform 304 includesmultiple virtual warehouses, including virtual warehouse 1, virtualwarehouse 2, and virtual warehouse n. Each virtual warehouse includesmultiple execution nodes that each include a data cache and a processor.The virtual warehouses can execute multiple tasks in parallel by usingthe multiple execution nodes. As discussed herein, execution platform304 can add new virtual warehouses and drop existing virtual warehousesin real-time based on the current processing needs of the systems andusers. This flexibility allows the execution platform 304 to quicklydeploy large amounts of computing resources when needed without beingforced to continue paying for those computing resources when they are nolonger needed. All virtual warehouses can access data from any datastorage device (e.g., any storage device in shared database storage308). Although each virtual warehouse shown in FIG. 5 includes threeexecution nodes, a particular virtual warehouse may include any numberof execution nodes. Further, the number of execution nodes in a virtualwarehouse is dynamic, such that new execution nodes are created whenadditional demand is present, and existing execution nodes are deletedwhen they are no longer necessary.

Each virtual warehouse is capable of accessing any of the data storagedevices 310 a-310 n shown in FIG. 3. Thus, the virtual warehouses arenot necessarily assigned to a specific data storage device and, instead,can access data from any of the data storage devices 310 a-310 n withinthe shared database storage 308. Similarly, each of the execution nodesshown in FIG. 5 can access data from any of the data storage devices 310a-310 n. In some embodiments, a particular virtual warehouse or aparticular execution node may be temporarily assigned to a specific datastorage device, but the virtual warehouse or execution node may lateraccess data from any other data storage device.

In the example of FIG. 5, virtual warehouse 1 includes three executionnodes 502 a, 502 b, and 502 n. Execution node 502 a includes a cache 504b and a processor 506 a. Execution node 502 b includes a cache 504 b anda processor 506 b. Execution node 502 n includes a cache 504 n and aprocessor 506 n. Each execution node 502 a, 502 b, and 502 n isassociated with processing one or more data storage and/or dataretrieval tasks. For example, a virtual warehouse may handle datastorage and data retrieval tasks associated with an internal service,such as a clustering service, a materialized view refresh service, afile compaction service, a storage procedure service, or a file upgradeservice. In other implementations, a particular virtual warehouse mayhandle data storage and data retrieval tasks associated with aparticular data storage system or a particular category of data.

Similar to virtual warehouse 1 discussed above, virtual warehouse 2includes three execution nodes 512 a, 512 b, and 512 n. Execution node512 a includes a cache 514 a and a processor 516 a. Execution node 512 bincludes a cache 514 b and a processor 516 b. Execution node 512 nincludes a cache 514 n and a processor 516 n. Additionally, virtualwarehouse 3 includes three execution nodes 522 a, 522 b, and 522 n.Execution node 522 a includes a cache 524 a and a processor 526 a.Execution node 522 b includes a cache 524 b and a processor 526 b.Execution node 522 n includes a cache 524 n and a processor 526 n.

In some embodiments, the execution nodes shown in FIG. 5 are statelesswith respect to the data the execution nodes are caching. For example,these execution nodes do not store or otherwise maintain stateinformation about the execution node, or the data being cached by aparticular execution node. Thus, in the event of an execution nodefailure, the failed node can be transparently replaced by another node.Since there is no state information associated with the failed executionnode, the new (replacement) execution node can easily replace the failednode without concern for recreating a particular state.

Although the execution nodes shown in FIG. 5 each include one data cacheand one processor, alternative embodiments may include execution nodescontaining any number of processors and any number of caches.Additionally, the caches may vary in size among the different executionnodes. The caches shown in FIG. 5 store, in the local execution node,data that was retrieved from one or more data storage devices in theshared database storage 308. Thus, the caches reduce or eliminate thebottleneck problems occurring in platforms that consistently retrievedata from remote storage systems. Instead of repeatedly accessing datafrom the remote storage devices, the systems and methods describedherein access data from the caches in the execution nodes which issignificantly faster and avoids the bottleneck problem discussed above.In some embodiments, the caches are implemented using high-speed memorydevices that provide fast access to the cached data. Each cache canstore data from any of the storage devices in the shared databasestorage 308.

Further, the cache resources and computing resources may vary betweendifferent execution nodes. For example, one execution node may containsignificant computing resources and minimal cache resources, making theexecution node useful for tasks that require significant computingresources. Another execution node may contain significant cacheresources and minimal computing resources, making this execution nodeuseful for tasks that require caching of large amounts of data. Yetanother execution node may contain cache resources providing fasterinput-output operations, useful for tasks that require fast scanning oflarge amounts of data. In some embodiments, the cache resources andcomputing resources associated with a particular execution node aredetermined when the execution node is created, based on the expectedtasks to be performed by the execution node.

Additionally, the cache resources and computing resources associatedwith a particular execution node may change over time based on changingtasks performed by the execution node. For example, an execution nodemay be assigned more processing resources if the tasks performed by theexecution node become more processor-intensive. Similarly, an executionnode may be assigned more cache resources if the tasks performed by theexecution node require a larger cache capacity.

Although virtual warehouses 1, 2, and n are associated with the sameexecution platform 304, the virtual warehouses may be implemented usingmultiple computing systems at multiple geographic locations. Forexample, virtual warehouse 1 can be implemented by a computing system ata first geographic location, while virtual warehouses 2 and n areimplemented by another computing system at a second geographic location.In some embodiments, these different computing systems are cloud-basedcomputing systems maintained by one or more different entities.

Additionally, each virtual warehouse is shown in FIG. 5 as havingmultiple execution nodes. The multiple execution nodes associated witheach virtual warehouse may be implemented using multiple computingsystems at multiple geographic locations. For example, an instance ofvirtual warehouse 1 implements execution nodes 502 a and 502 b on onecomputing platform at a geographic location and implements executionnode 502 n at a different computing platform at another geographiclocation. Selecting particular computing systems to implement anexecution node may depend on various factors, such as the level ofresources needed for a particular execution node (e.g., processingresource requirements and cache requirements), the resources availableat particular computing systems, communication capabilities of networkswithin a geographic location or between geographic locations, and whichcomputing systems are already implementing other execution nodes in thevirtual warehouse.

Execution platform 304 is also fault tolerant. For example, if onevirtual warehouse fails, that virtual warehouse is quickly replaced witha different virtual warehouse at a different geographic location.

A particular execution platform 304 may include any number of virtualwarehouses. Additionally, the number of virtual warehouses in aparticular execution platform is dynamic, such that new virtualwarehouses are created when additional processing and/or cachingresources are needed. Similarly, existing virtual warehouses may bedeleted when the resources associated with the virtual warehouse are nolonger necessary.

In some embodiments, the virtual warehouses may operate on the same datain the shared database storage 308 but each virtual warehouse has itsown execution nodes with independent processing and caching resources.This configuration allows requests on different virtual warehouses to beprocessed independently and with no interference between the requests.This independent processing, combined with the ability to dynamicallyadd and remove virtual warehouses, supports the addition of newprocessing capacity for new users without impacting the performanceobserved by the existing users.

The execution platform 304 may execute queries against an externaltable, where the external table is not managed by the database platformbut can be read by the database platform. The execution platform 302 mayexecute such queries by communicating with the external table andreading data directly from the external table.

FIG. 6 is a schematic block diagram of an database platform 102. Thedatabase platform 102 includes a data lake notification component 602,an external table notification component 604, an external table metadatacomponent 606, a metadata refresh component 608, a materialized viewgeneration component 610, a materialized view refresh component 612, aquery component 614, and a security component 616. The database platform102 may be in communication with a data lake 620 that may include anexternal table 622. The database platform 102 may further be incommunication with one or more “internal” tables 626 that are managed bya database platform 102. The database platform 102 is in communicationwith and can read and write to metadata 624 that includes informationabout one or more of the data in the data lake 620, the external table622, and/or the internal table 626.

The data lake notification component 602 receives a notification thatdata has been added, deleted, or modified in a data lake 620. The datalake notification component 602 may alternatively be referred to as adata lake notification listener component. The data lake notificationcomponent 602 may proactively query the data lake 620 to determine ifany data has been added, deleted, or modified since the last time thedatabase platform read data in the data lake. The data lake notificationcomponent 602 may receive a notification from the data lake 620 that isautomatically or manually generated by the data lake provider or theclient.

In an embodiment, the data lake notification component 602 is configuredto detect an update to the source directory within the data lake 620,wherein the external table 622 is based on the source directory. In anembodiment, the data lake notification component 602 receives anindication that a file has been added, modified, or removed from thesource directory. In an embodiment, the data lake notification component602 is in communication with a simple queue service (“SQS”) message.This communication may be set up by the client or a system administratorwhen the external table 622 is generated. The SQS queue id (ARN) may besupplied to the data lake 620 back such that push notifications will begenerated via SQS each time a file is added, updated, or deleted fromthe data lake 620. In an embodiment, the data lake notificationcomponent 602 receives a SQL command to register and deregister a filein an external table 622. The SQL command may be received from a clientand may indicate that metadata should be refreshed based on the currentstate of the external table 622 location.

The external table notification component 604 receives a notificationthat data has been added, deleted, or modified in an external table 622.The external table notification component 604 may alternatively bereferred to as an external table notification listener component. Theexternal table notification component 604 may proactively query theexternal table 622 to determine if any data has been added, deleted, ormodified since the last time the database platform read data in theexternal table 622. The external table notification component 602 mayreceive a notification from the external table 622 or the data lake 620that is automatically or manually generated by the data lake provider(i.e., also the external table provider) or the client. The externaltable 622 may constitute a source table that is stored within the datalake 620. The source table may be defined by the client and the clientmay add, delete, or modify data stored in the source table. The clientmay create an external table 622 with the database platform by grantingthe database platform read permissions to the source table within thedata lake 620.

The external table metadata component 606 generates, manages, andrefreshes metadata 624 that includes information about data stored inthe external table 622. The metadata 624 may be stored according to themetadata structures disclosed in FIG. 10 and FIGS. 11A-11C. The externaltable metadata component 606 may read the data stored in the externaltable 622 to determine the metadata 624. The external table metadatacomponent 606 may determine the metadata 624 based on how data isorganized within the external table 622. The client may organize data inthe external table 622 according to a hierarchical structure, and thedetails of the hierarchical structure may be provided to the databaseplatform. The hierarchical structure of organization for the externaltable 622 includes an indication of how folders and subfolders of datain the external table 622 are organized. In an example, a hierarchicalstructure may indicate that one folder includes all transactions for acertain year and that subfolders within that folder include alltransactions for each month of that year. The data in the external table622 may be partitioned into partitions according to some other parametersuch as location or name. The client uploads files to a specific folderin the external table 622, and the location of the file (i.e., whichfolder the file is uploaded to) guides the external table metadatacomponent 606 in generating the metadata 624 about the external table622.

The metadata refresh component 608 refreshes the metadata 624 about theexternal table 622 in response to changes that are made to the externaltable 622. The metadata 624 for the external table 622 is refreshed whendata is added, deleted, or updated in the external table 622.

The materialized view generation component 610 generates one or morematerialized views over the external table 622. The materialized viewgeneration component 610 may generate a materialized view based on aquery received from the client. The client may manually request that amaterialized view be generated for a certain dataset in the externaltable 622. The database platform may automatically determine that amaterialized view should be generated for a certain dataset in theexternal table 622 if that dataset applies to a query that is frequentlyrequested by the client. The materialized view is stored with thedatabase platform 102 and may be stored in cache storage or in theshared database storage 308. The materialized view may be stored incache storage associated with a node of the execution platform 304.

The materialized view generation component 610 generates thematerialized view based on the external table 622. The materialized viewgeneration component 610 may generate the materialized view to have itsown domain and have the characteristics of both a table and a view withadditional information linked to the external table 622 and versioninginformation related to the external table 622 and/or the sourcedirectory within the data lake 620. The materialized view is adeclarative specification of a persistent query result that isautomatically maintained and transparently utilized. The materializedview is a database object that includes the results of a persistentquery result on the external table 622 and/or the source directorywithin the data lake 620. The materialized view may be a local copy ofdata located remotely, it may be a subset of rows and/or columns of thesource directory or join result, or it may be a summary using anaggregate function. The materialized view generation component 610 maybe configured to generate the materialized view by caching the resultsof a query by the process of materialization such that the cached queryresult is stored as a concrete “materialized” table that may be updatedfrom the original external table 622 and/or source directory within thedata lake 620. The materialized view may provide improved performance ofdatabase queries on large select, join, or aggregate statements. Thematerialized view provides additional storage that is small compared tothe source directory, the materialized view may be automaticallymaintained, and the materialized view may be transparently used withoutchanging an existing workload on the external table 622 and/or thesource directory within the data lake 620. The materialized viewincludes additional information linked to its management, including asource directory identifier, a set of partitions materialized since thelast refresh version of the materialized view, and a set of partitionsremoved since a last compact version of the materialized view.

In an embodiment, the materialized view generation component 610 storeswithin the materialized view the same information as for tables e.g.stage information and for views e.g. view definitions. Additionally, thematerialized view generation component 610 stores a source directoryidentifier. The source directory identifier is tagged to thematerialized view during compilation to indicate the source directorythat will be utilized for maintenance and incremental updating of thematerialized view. The materialized view generation component 610further stores an indication of a type of materialized view, wherein thetype of materialized view indicates an enumerated type that is utilizedto determine the scope of a materialized view (e.g. projection, summary,synopses, join, etc.). In addition, the materialized view generationcomponent 610 may include information specific to DML versioning that isadded to the table version associated with the materialized view. Thematerialized view may be tagged with a time of a prior refresh and atime of a prior compaction of the materialized view.

The materialized view refresh component 612 refreshes a materializedview that is generated over an external table. The materialized viewrefresh component 612 may further refresh materialized views that aregenerated over internals table 626 that are managed by the databaseplatform 102. The materialized view refresh component 612 refreshes amaterialized view over an external table when data is added, deleted, orupdated in the external table. The materialized view may be refreshedafter a threshold number of modifications have been made to the externaltable, after a threshold time duration has passed since the lastrefresh, after each update to the external table, or based on anysuitable metric. In an embodiment, the client may define when and howoften the materialized view is refreshed and/or may define parametersfor an acceptable amount of inconsistency or staleness between theexternal table and the materialized view.

The materialized view refresh component 612 may maintain a refreshconstruct that indicates the set of partitions to insert into thematerialized view by pulling the log of added partitions that have beenadded to the source directory of the data lake 620 since the lastrefresh of the materialized view. The materialized view refreshcomponent 612 maintains a compact construct that indicates the set ofpartitions to remove from the materialized view by pulling the log ofpartitions that have been removed from the source directory since thelast compaction of the materialized view.

The materialized view refresh component 612 is configured to perform arefresh on the materialized view by adding one or more partitions or newrows to the materialized view. The materialized view refresh component612 receives an indication from the data lake notification component 602and/or the external table notification component 604 that a partitionhas been inserted into the source directory of the data lake 620 sincethe last refresh of the materialized view. The materialized view refreshcomponent 612 then inserts that partition into the materialized view.The materialized view refresh component 612 may receive a single sourcedirectory identifier for each materialized view that it generates. In anembodiment, the materialized view refresh component 612 is manuallytriggered, enabled, or disabled by an account issuing a command. In anembodiment, the materialized view refresh component 612 automaticallyupdates the materialized view when the materialized view refreshcomponent 612 is enabled.

In an embodiment, the materialized view refresh component 612 carriesout an insert command. The materialized view refresh component 612receives a log of new partitions that have been added to the sourcedirectory of the data lake 620 since a prior refresh of the externaltable and/or the materialized view. The materialized view refreshcomponent 612 inserts the new partitions into the materialized view andmay be configured to group the new partitions in a column of thematerialized view and order the new partitions in the materialized view.In an embodiment, the metadata for the new partition is consistentbetween the source directory of the data lake 620 and the external table622.

The query component 614 determines a query plan to execute queries overan external table. The resource manager 302 may receive a query from aclient. The query may include a Structured Query Language (SQL) queryhaving certain parameters. The query component 614 reads the sharedmetadata 312 to determine where the applicable files for responding tothe query are stored. The applicable files may be stored in “internal”data stored in the shared database storage 308 managed by the databaseplatform 102, may be stored in cache storage in the execution platform304, may be stored in the external table 106, may be stored inmaterialized views, or may be stored in any combination of the above.The query component 614 determines the files that are necessary torespond to the query and further determines where those files arelocated across the shared database storage 308, cache storage in theexecution platform 304, the external table 106, and/or materializedviews. The query component 614 generates a query plan that includes oneor more discrete subtasks to be executed to respond to the query. Thequery component 614 assigns the one or more discrete subtasks todifferent nodes of the execution platform 304 for processing. The querycomponent 614 assigns the one or more discrete subtasks based on wherethe files are stored. For example, if a certain execution node hascached a materialized view that includes data necessary to respond tothe query, that execution node will be assigned a subtask pertaining tothat data. Further for example, if a certain execution node has cached aportion of the external table or the “internal” data managed by thedatabase platform, that execution node will be assigned a subtaskpertaining to the data. The query is executed by the execution platformand may be returned to the client by the resource manager 302.

The security component 616 manages permissions for the data lake 620,the external table 106, the database platform 102, the shared databasestorage 308, and/or materialized views. The security component 616carries out one or more security protocols to protect data in any of thesource directory of the data lake 620, the external table 622, themetadata 624, the internal table 626, and/or the materialized view. Inan embodiment, the security component 616 provides an account setting toan account associated with the source directory such that the accountmay request an external table and/or a materialized view to be generatedbased on the source directory. As such, the security component 616 mayfurther provide an account setting to a second account associated withthe source directory such that the second account does not havepermission to request an external table or a materialized view. In anembodiment, the security component 616 encrypts data in the sourcedirectory, the external table, or the materialized view. In anembodiment, the security component 616 rekeys or re-encrypts data in thesource directory, the external table, or the materialized view atpredetermined time periods, upon client request, and/or when data isupdated.

In an embodiment, the work performed by the database platform 102 isperformed in the background and may occur in a continuous fashion. In anembodiment, as soon as a materialized view is generated based on anexternal table 622, the database platform 102 is configured to maintainfreshness of the materialized view. In an embodiment, a maximum budgetfor materialized view maintenance may be altered by a client. The clientmay provide a significant budget for maintenance operations of certainmaterialized views but provide a smaller budget for other materializedviews. The client may have access to a plurality of priority settingsfor maintenance operations of various materialized views. The client mayfurther receive information indicating how frequently and howefficiently a materialized view is refreshed or compacted.

FIG. 7 is a schematic block diagram of an example process flow 700 forquerying over an external table. The process flow 700 may be carried outby a resource manager 302 and an execution platform 304 as describedherein. The client 118 may include an account associated with theresource manager 302 (i.e., associated with the database platform 102)and further associated with a data lake 104 that stores the data for theexternal table 106. The client 118 provides a query to the resourcemanager 302. The query may include a Structured Query Language (SQL)statement. The resource manager 302 receives the query from the clientaccount at 702 and the query includes one or more query predicates. Theresource manager 302 references metadata to identify the partitions thatare necessary to respond to the query at 704 by determining whichpartitions within the external table and/or the shared database storage308 include data that satisfies the one or more query predicates. Theresource manager 302 generates a query plan that includes a plurality ofdiscrete subtasks that need to be processed to respond to the query at706. The resource manager 302 assigns the plurality of discrete subtasksto one or more nodes of an execution platform at 708.

The resource manager 302 may assign the plurality of discrete subtasksbased on metadata and further based on where the identified partitionsare stored. For example, the resource manager 302 may reference metadatato learn that a first identified partition is stored in cache storage ofa first execution node. The resource manager 302 may then assign thefirst identified partition to the first execution node to hasten queryresponse time and lessen the processing load to avoid retrieving thepartition from an external table or the shared database storage 308.Further for example, the resource manager 302 may determine that oneidentified partition is referenced in a materialized view. The resourcemanager 302 may assign an execution node to read the materialized viewto execute one of the discrete subtasks of the query plan.

In the process flow 700, an example execution node 306 of the executionplatform 304 has access to the external table 106. The execution node306 receives an instruction from the resource manager 302 to execute adiscrete subtask of the query plan at 710, where the discrete subtaskrequires the execution node 306 to read data in the external table 106.The execution node 306 access the external table at 712 to read one ormore partitions in the external table that are specified in the discretesubtask. The execution node 306 process the task at 714 by reading oneor more partitions in the external table and filtering rows that do notsatisfy the query predicates to return only those rows that satisfy thequery predicates.

In an embodiment, referencing metadata at 704 to identify partitionsnecessary to respond to the query includes referencing shared metadata312 to identifying partitions in the external table and further toidentify partitions in internal database storage (such as cache storageand/or data in the shared database storage 308) that are necessary torespond to the query. Referencing the metadata at 704 may furtherinclude determining whether a materialized view has been generated overany data in the external table or an internal table. In the event theexternal table and/or the internal table has been modified since a mostrecent refresh of the corresponding materialized view, the query planmay include supplementing the materialized view with data in theexternal table or the internal table. This may be accomplished byreading metadata for the materialized view to determine a timestamp whenthe materialized view was refreshed, and further by reading metadata forthe corresponding external table and/or internal table to determinetimestamps when recent modifications were made to the external tableand/or the internal table. The timestamps may be compared to determininghow many modifications and which modifications were made to the externaltable and/or the internal table since the most recent refresh of thecorresponding materialized view. Supplementing the materialized view mayfurther be accomplished by reading metadata to determine whatmodifications were made at certain timestamps. The timestamps mayinclude information about what modifications were made, for example whatData Manipulation Language (DML) commands were executed, such as update,delete, and/or merge commands. The materialized view may be supplementedbased on this metadata to determine which rows in the materialized vieware stale with respect to the external table and/or the internal table.Those rows that are determined to be stale will not be read from thematerialized view but will instead be read from the internal tableand/or the external table.

FIG. 8 illustrates a schematic block diagram of an example process flow800 for incremental updating of a materialized view. The process flow800 may be carried out by any suitable computing device, including forexample a resource manager 302 and/or an database platform 102 andspecifically a materialized view generation component 610 and/or amaterialized view refresh component 612. The process flow 800 includesgenerating a materialized view at 802 that is based on a source table(in an embodiment as disclosed herein, the source table may be anexternal table that is based on a data lake source directory, forexample). The process flow 800 includes updating the source table at 804which may include inserting a partition at 806 and/or removing apartition at 808. The process flow includes querying the materializedview and the source table at 810 to detect any updates to the sourcetable that are not reflected in the materialized view. The process flow800 includes applying the update to the materialized view at 812.

In the example illustrated in FIG. 8, the materialized view is generatedat 802 by scanning a source table. The source table includes the dataset{1 2 3 4 5 6} as illustrated. The corresponding materialized viewincludes the data sets [1(1 2 3)] and [2(4 5 6)] that may indicatepartitions in the database, where the partitions are immutable storageobjects in the database.

In the example illustrated in FIG. 8, the source table is updated at 804(see Δ1) by adding (+7) and removing (−2). Two partitions are insertedinto the source table at 806 (see Δ2) by adding (+8) and adding (+9).Two partitions are removed from the source table at 808 (see Δ3) byremoving (−1) and (−3). As illustrated in Δ (delta), the overall updateto the source table includes {+7 +8 +9 −1 −2 −3}, which includes each ofthe various updates (see Δ1, Δ2, and Δ3) that are made on the sourcetable.

In the example illustrated in FIG. 8, the materialized view and thesource table are queried at 810. The query 810 includes merging thematerialized view and the source table. The source table is scanned andpartitions {7 8 9} are detected in the source table and those partitionsare not detected in the materialized view. The materialized view isscanned and the partition (1 2 3) is detected in the materialized viewand it is not detected in the source table.

In the example illustrated in FIG. 8, the update is applied to thematerialized view at 812. The materialized view is scanned, and thesystem detects the partitions [1(1 2 3)] and [2(4 5 6)], and the systemdetects that partition [1(1 2 3)] is present in the materialized viewand should be removed. The system deletes the partition [1(1 2 3)] suchthat the partition [2(4 5 6)] remains. The system scans the source tableand discovers the partition {7 8 9} and inserts that partition into thematerialized view such that the materialized view includes thepartitions [2(4 5 6)] and [3(7 8 9)].

FIG. 9 illustrates a schematic block diagram of an example process flow900 for updating of a materialized view over an external table. Theprocess flow 900 may be carried out by any suitable computing device,including for example a resource manager 302 and/or an database platform102 and specifically a materialized view generation component 610 and/ora materialized view refresh component 612 of the database platform 102.The materialized view is generated over an external table such that thematerialized view is generated, refreshed, and stored with a databaseplatform 102 and the source table for the materialized view (i.e., theexternal table 106) is not managed, stored, or updated by the databaseplatform. The materialized view is internal to the database platform 102and the source table is external to the database platform 102. Thematerialized view may be stored in persistent storage such as the shareddatabase storage 308 or it may be stored in cache storage, such as cachestorage within an execution node 306 of the execution platform 304. Thesource table for the materialized view (i.e. the external table) may bestored in persistent storage in a data lake.

The process flow 900 includes an initial refresh of the materializedview at 902. The initial refresh at 902 includes scanning the externaltable (the source table for the materialized view) and refreshing thematerialized view. The process flow 900 includes updates being made tothe external table at 904. The updates may be initiated by a client thatis in communication with a data lake that supports the external table,and the updates may be implemented by a provider of the data lake. Inthe example in FIG. 9, the updates are made to micro-partition 5 (MIP5)and micro-partition 6 (MP6). The updates may include modifying an entryin one or more rows of MP5 and MP6.

The process flow 900 includes inserts being made to the external tableat 906. The inserts may be initiated by a client that is incommunication with a data lake that supports the external table, and theinserts may be implemented by a provider of the data lake. In theexample in FIG. 9, the inserts performed at 906 are executed onmicro-partition 1 (MP1) and micro-partition 2 (MP2). The inserts mayinclude one or more rows being added to MP1 or MP2. The inserts mayinclude adding MP1 or MP2 if one or more of those partitions did notpreviously exist.

The process flow 900 includes deletes being executed on the externaltable at 908. The deletes may be initiated by a client that is incommunication with a data lake that supports the external table, anddeletes may be implemented by a provider of the data lake. In theexample in FIG. 9, the deletes performed at 908 are executed onmicro-partition 3 (MP3) and micro-partition 4 (MP4). The deletes mayinclude one or more rows being removed from MP3 and/or MP4. The deletesmay include MP3 and/or MP4 being removed entirely.

The process flow 900 includes applying changes made to the externaltable (i.e. the source table for the materialized view) to thematerialized view at 910. Applying the changes at 910 includesgenerating a new MP5 and a new MP6 having the updated rows according tothe updates that were made to the external table at 904. Applying theupdates includes removing and replacing the modified rows. Applying thechanges at 910 includes generating a new MP1 and a new MP2 havinginserted rows according to the inserts that were made to the externaltable at 906. Applying the changes at 910 includes generating a new MP3and a new MP4 with the deleted rows being removed according to thedeletes that were made to the external table at 908. The process flow900 includes querying the materialized view at 912. Because thematerialized view has been refreshed to reflect updates made to theexternal table, queries may be executed over the materialized viewrather than the external table itself to hasten query response times.

FIG. 10 is a schematic diagram of a data structure 1000 for storage ofdatabase metadata, including in persistent storage and cache storage.The data structure 1000 may be applied to the storage of metadata forexternal tables that are not managed by a database platform 102 and/orfor internal tables that are managed and stored by the database platform102. The data structure 1000 includes cumulative table metadata 1002including information about a table. The table may be an external tableor an internal table. The table may include a plurality of files orpartitions that may each include a number of columns and rows storingdata. The cumulative table metadata 1002 includes global informationabout the table and may include summary information stored in each of aplurality of grouping expression properties 1014 a, 1014 b, 1014 c, and1014 d (may be collectively referenced herein as “1014”). The groupingexpression properties 1014 include aggregated partition statistics,cumulative column properties, and so forth about a partition 1006 or acollection of partitions of the table. It should be appreciated that thepartitions 1006 illustrated in FIG. 10 may each contain a differentsubset of the data stored in the table and may include the same columnsor may include different columns storing different types of information.The partitions 1006 of the table each include one or more columns andmay each have the same types of columns or different types of columns.An expression property may be stored for each column of each partition1006 of the table, or for a collection of partitions 1006 of the tableas illustrated in FIG. 10. The data structure 1000 includes partitionstatistics 1004 for each partition 1006 of the table (the partitionstatistics 1004 may alternatively be referred to herein as “partitionexpression properties”). The partition statistics 1004 may include aminimum/maximum data point for the corresponding partition 1006, a typeof data stored in the corresponding partition 1006, a partitionstructure of the corresponding partition 1006, and so forth. Asillustrated in FIG. 10, a column 1 expression property 1008 is storedfor the first column in each of the different partitions 1006. Further,a column 2 expression property 1010 is stored for the second column ineach of the different partitions 1006. Further, a column 3 expressionproperty 1012 is stored for the third column in each of the differentpartitions. It should be appreciated that each of the partitions 1006may include any suitable number of columns, and that an expressionproperty may be stored for each of the columns, or for any suitablenumber of the columns, stored in each partition of the table. The column1 expression properties 1008, the column 2 expression properties 1010,and the column 3 expression properties 1012, along with any additionalcolumn expression properties that may be included as deemed appropriate,may be stored as part of a metadata partition. A metadata partition maybe persisted in immutable storage and the grouping expression properties1014 may also be stored within a metadata partition in immutablestorage. A metadata manager may maintain all metadata partitions,including metadata partitions comprising the grouping expressionproperties 1014, and partition statistics 1004, and/or the columnexpression properties 1008-1012.

The cumulative table metadata 1002 includes global information about allpartitions within the applicable table. For example, the cumulativetable metadata 1002 may include a global minimum and global maximum forthe entire table, which may include millions or even hundreds ofmillions of partitions. The cumulative table metadata 1002 may includeany suitable information about the data stored in the table, including,for example, minimum/maximum values, null count, a summary of thedatabase data collectively stored across the table, a type of datastored across the table, a distinct for the data stored in the table,and so forth.

The grouping expression properties 1014 a-1014 d include informationabout database data stored in an associated grouping of partitions. Forexample, an example grouping expression property is associated withpartitions numbered 3040 thru 3090 such that the example groupingexpression property is associated with fifty different partitions. Theexample grouping expression property includes information about thosefifty different partitions. A grouping expression property 1014 mayinclude any suitable information about the partitions with which it isassociated. For example, a grouping expression property 1014 may includea global minimum/maximum for the collective set of partitions, aminimum/maximum for each of the partitions within the grouping, a globalnull count, a null count for each of the partitions within the grouping,a global summary of data collectively stored across the grouping ofpartitions, a summary of data stored in each of the partitions in thegrouping, and so forth. The grouping expression property 1014 mayinclude global information for all partitions within the grouping ofpartitions that is associated with the grouping expression property1014, and it may further include information specific to each of thepartitions within the associated grouping.

The metadata structure disclosed in FIG. 10 provides increasedgranularity in cumulative table metadata 1002 over other embodiments ofdatabase metadata structures. The grouping expression properties 1014provide valuable global metadata pertaining to a collection ofpartitions 1006 of the database. Further, each of the columnarexpression properties 1008, 1010, 1012 provide valuable informationabout a column of a partition 1006 of the table.

The metadata structures disclosed herein, including the data structure1000 shown in FIG. 10, increases efficiency when responding to databasequeries. A database query may request any collection of data from thedatabase and/or an external table and may be used for creating advancedanalyses and metrics about the database data and/or the data stored inan external table. Some queries, particularly for a very large database,external table, and/or data lake, can be extremely costly to run both intime and computing resources. When it is necessary to scan metadataand/or database data for each file or partition of each table of adatabase, it can take many minutes or even hours to respond to a query.In certain implementations this may not be an acceptable use ofcomputing resources. The data structure 1000 disclosed herein providesincreased metadata granularity and enables multi-level pruning ofdatabase data and/or data stored in an external table. Duringcompilation and optimization of a query on the database, a processor mayscan the cumulative table metadata 1002 to determine if the tableincludes information pertaining to the query. In response todetermining, based on the cumulative table metadata 1002, that the tableincludes information pertaining to the query, the processor may scaneach of the grouping expression properties 1014 to determine whichgrouping of partitions of the table include information pertaining tothe query. In response to determining, based on a first cumulativeexpression property, that a first grouping of partitions does notinclude information pertaining to the query, the processor maydiscontinue database scanning of that first grouping of partitions. Inresponse to determining, based on a second cumulative expressionproperty, that a second grouping of partitions includes informationpertaining to the query, the processor may proceed to scan expressionproperties for that second grouping of partitions. The processor mayefficiently determine which partitions include pertinent data and whichcolumns of which partitions include pertinent data. The processor mayproceed to scan only the relevant column(s) and partition(s) thatinclude information relevant to a database query. This provides a costefficient means for responding to a database query by way of multi-levelpruning based on multi-level table metadata.

Further to increase the cost efficiency of database queries, a resourcemanager (may also be referred to as a “global services”) may store thecumulative table metadata 1002 in a cache for faster retrieval. Metadatafor the database may be stored in a metadata store separate andindependent of a plurality of shared storage devices collectivelystoring database data. In a different embodiment, metadata for thedatabase may be stored within the plurality of shared storage devicescollectively storing database data. In various embodiments, metadata maybe stored in metadata-specific partitions that do not include databasedata, and/or may be stored within partitions that also include databasedata. The metadata may be stored across disk storage, such as theplurality of shared storage devices, and it may also be stored in cachewithin the resource manager.

FIGS. 11A-11C illustrate the generation of cumulative table metadata1102 across multiple versions of a table. The table may be storedinternally to a database platform or may be an external table. Themetadata versioning as illustrated in FIGS. 11A-11C may be executed byany suitable computing device, including an database platform 102 andspecifically an external table metadata component 606 and/or a metadatarefresh component 608. The refreshing and versioning of table metadatamay be prompted by receiving a notification from a data lake that asource table supporting an external table has been modified. Themetadata structure depicted in FIGS. 11A-11C follows the data structure1000 disclosed in FIG. 10. Accordingly, there is cumulative tablemetadata 1102 including global information about the table, along withgrouping expression properties 1104 for each grouping of tablepartitions. Further, the table includes a plurality of partitions eachserving as an immutable storage device for storing database data thatcannot be updated in-place.

FIG. 11A illustrates metadata for a first version (V1) of the table. Thetable has version one cumulative table metadata 1102 a and version onegrouping expression properties 1104 a each including global informationabout a grouping of partitions 1106 of the table. Version 1 of thepartitions, including A1, B1, C1, D1, E1, F1, G1, and H1 are illustratedin FIG. 11A. The table includes a plurality of partitions each storing asubset of the database data in the table.

FIG. 11B illustrates metadata for a second version (V2) of the tablethat includes a second version of the partitions, including A2, B2, C2,D2, E2, F2, G2, H2, I2, J2, K2, and L2. The greyed partitions (C2, E2,and G2) illustrate partitions that were deleted by a transactionexecution on the table. The new partitions (I2, J2, K2, and L2)illustrate partitions that were added to the table by a transaction thatwas executed on the table. The non-greyed existing partitions (A2, B2,D2, F2, and H2) illustrate partitions that were not modified by anytransaction executed on the table. As shown in FIG. 11B, a version twocumulative table metadata 1102 b is generated in response to a newversion of the table being generated by one or more transactions beingexecuted on the table. Further, new version two grouping expressionproperties 1104 b are generated in response to the new version of thetable being generated by the one or more transactions executed on thetable. In an embodiment, when partitions are deleted from a grouping ofpartitions, but no new partitions were added to the grouping, thegrouping expression properties 1104 b may not need to be recomputed. Theminimum value, maximum value, number of nulls, and number of distinctvalues may now be overestimated, but may still be considered safe by thedatabase client. The overestimated values may lead an optimizer tosuboptimal decisions but not to wrong query results. In this embodiment,the grouping expression properties 1104 b may still be recomputed forthe sake of optimizer efficiency. In the example illustrated in FIG.11B, after partitions C2, E2, and G2 are deleted, the existing groupingexpression properties 1104 b for the first and second grouping are stillsafe to use. These grouping expression properties 1104 b may not berecomputed or may be recomputed in a later phase. For each new versionof the table, the grouping expression properties may be computed forgroupings of newly added partitions, and this may lead to efficientcalculation of global grouping expression properties. Each of thepartitions 1106 of the table constitutes an immutable storage devicethat cannot be updated in-place. Therefore, in response to a transactionbeing executed on the table, such as a DML command, a new partition isgenerated to reflect the transaction and replace the prior partition.

FIG. 11C illustrates metadata for a third version (V3) of the table. Asillustrated, new partitions have been added to the table in versionthree, including partitions M3, N3, 03, and P3. An updated version threecumulative table metadata 1102 c provides global information about thetable, and updated version three grouping expression properties 1104 ceach provide global information about their associated groupings ofversion three partitions.

FIG. 12 is a schematic flow chart diagram of a method 1200 formaintaining an external table in a database system. The method 1200 maybe performed by any suitable computing device such as a resource manager302 or database platform 102 as disclosed herein.

The method 1200 begins and the computing device receives at 1202 readaccess to a source directory in a data storage platform. The computingdevice is associated with a database platform that is separate from thedata storage platform. The computing device defines at 1204 an externaltable based on the source directory. The computing device connects at1206 the database platform to the external table such that the databaseplatform has read access for the external table and does not have writeaccess for the external table. The computing device generates at 1208metadata for the external table, the metadata including informationabout data stored in the external table. The computing device receivesat 1210 a notification that a modification has been made to the sourcedirectory, the modification including one or more of an insert, adelete, or an update. The computing device refreshes at 1212 themetadata for the external table in response to the modification beingmade to the source directory.

FIG. 13 is a schematic flow chart diagram of a method 1300 for definingan external table in a database system. The method 1300 may be performedby any suitable computing device such as a resource manager 302 ordatabase platform 102 as disclosed herein.

The method 1300 begins and the computing device receives at 1302 anindication of a hierarchical structure in a source directory, thehierarchical structure defining folders and subfolders for data in thesource directory. The computing device receives at 1304 an indication ofa partitioning structure for data in the source directory. The computingdevice defines at 1306 partitions in an external table based on wherefiles are uploaded within the hierarchical structure and further basedon the partitioning structure.

FIG. 14 is a schematic flow chart diagram of a method 1400 for queryingan external table in a database system. The method 1400 may be performedby any suitable computing device such as a resource manager 302 ordatabase platform 102 as disclosed herein.

The method 1400 begins and the computing device connects at 1402 adatabase platform to an external table such that the database platformhas read access for the external table and does not have write accessfor the external table. The computing device receives at 1404 a queryincluding a predicate, the query directed at least to data in theexternal table. The computing device determines at 1406, based onmetadata, one or more partitions in the external table including datasatisfying the predicate. The computing device prunes at 1408, based onthe metadata, all partitions in the external table that do not includeany data satisfying the predicate. The computing device generates at1410 a query plan including a plurality of discrete subtasks. Thecomputing device assigns at 1412, based on the metadata, the pluralityof discrete subtasks to one or more nodes in an execution platform.

FIG. 15 is a schematic flow chart diagram of a method 1500 forgenerating a materialized view over an external table in a databasesystem. The method 1500 may be performed by any suitable computingdevice such as a resource manager 302 or database platform 102 asdisclosed herein.

The method 1500 begins and the computing device connects at 1502 adatabase platform to an external table such that the database platformhas read access for the external table and does not have write accessfor the external table. The computing device generates at 1504 amaterialized view over the external table. The computing device receivesat 1506 a notification that a modification has been made to the externaltable, the modification including one or more of an insert, a delete, oran update. The computing device, in response to the external table beingmodified, refreshes at 1508 the materialized view such that thematerialized view comprises an accurate representation of the externaltable.

FIG. 16 is a schematic flow chart diagram of a method 1600 for queryingan external table in a database system. The method 1600 may be performedby any suitable computing device such as a resource manager 302 ordatabase platform 102 as disclosed herein.

The method 1600 begins and the computing device connects at 1602 adatabase platform to an external table such that the database platformhas read access for the external table and does not have write accessfor the external table. The computing device receives at 1604 a queryincluding a predicate, the query directed at least to data in theexternal table. The computing device determines at 1606, based onmetadata, one or more partitions in the external table including datasatisfying the predicate. The computing device prunes at 1608, based onthe metadata, all partitions in the external table that do not includeany data satisfying the predicate. The computing device generates at1610 a query plan including a plurality of discrete subtasks. Thecomputing device reads at 1612 the metadata to determine whether a firstpartition is stored in a cache of any certain node of the executionplatform. The computing device, in response to the first partition beingstored in the cache of the certain node of the execution platform,assigns at 1614 the first partition to the certain node to be processedby the certain node.

FIG. 17 is a block diagram depicting an example computing device 1700.In some embodiments, computing device 1700 is used to implement one ormore of the systems and components discussed herein. Further, computingdevice 1700 may interact with any of the systems and componentsdescribed herein. Accordingly, computing device 1700 may be used toperform various procedures and tasks, such as those discussed herein.Computing device 1700 can function as a server, a client or any othercomputing entity. Computing device 1700 can be any of a wide variety ofcomputing devices, such as a desktop computer, a notebook computer, aserver computer, a handheld computer, a tablet, and the like.

Computing device 1700 includes one or more processor(s) 1702, one ormore memory device(s) 1704, one or more interface(s) 1706, one or moremass storage device(s) 1708, and one or more Input/Output (I/O)device(s) 1710, all of which are coupled to a bus 1712. Processor(s)1702 include one or more processors or controllers that executeinstructions stored in memory device(s) 1704 and/or mass storagedevice(s) 1708. Processor(s) 1702 may also include various types ofcomputer-readable media, such as cache memory.

Memory device(s) 1704 include various computer-readable media, such asvolatile memory (e.g., random access memory (RAM)) and/or nonvolatilememory (e.g., read-only memory (ROM)). Memory device(s) 1704 may alsoinclude rewritable ROM, such as Flash memory.

Mass storage device(s) 1708 include various computer readable media,such as magnetic tapes, magnetic disks, optical disks, solid statememory (e.g., Flash memory), and so forth. Various drives may also beincluded in mass storage device(s) 1708 to enable reading from and/orwriting to the various computer readable media. Mass storage device(s)1708 include removable media and/or non-removable media.

I/O device(s) 1710 include various devices that allow data and/or otherinformation to be input to or retrieved from computing device 1700.Example I/O device(s) 1710 include cursor control devices, keyboards,keypads, microphones, monitors or other display devices, speakers,printers, network interface cards, modems, lenses, CCDs or other imagecapture devices, and the like.

Interface(s) 1706 include various interfaces that allow computing device1700 to interact with other systems, devices, or computing environments.Example interface(s) 1706 include any number of different networkinterfaces, such as interfaces to local area networks (LANs), wide areanetworks (WANs), wireless networks, and the Internet.

Bus 1712 allows processor(s) 1702, memory device(s) 1704, interface(s)1706, mass storage device(s) 1708, and I/O device(s) 1710 to communicatewith one another, as well as other devices or components coupled to bus1712. Bus 912 represents one or more of several types of bus structures,such as a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.

For purposes of illustration, programs and other executable programcomponents are shown herein as discrete blocks, although it isunderstood that such programs and components may reside at various timesin different storage components of computing device 1700 and areexecuted by processor(s) 1702. Alternatively, the systems and proceduresdescribed herein can be implemented in hardware, or a combination ofhardware, software, and/or firmware. For example, one or moreapplication specific integrated circuits (ASICs) can be programmed tocarry out one or more of the systems and procedures described herein. Asused herein, the terms “module” or “component” are intended to conveythe implementation apparatus for accomplishing a process, such as byhardware, or a combination of hardware, software, and/or firmware, forthe purposes of performing all or parts of operations disclosed herein.The terms “module” or “component” are intended to convey independent inhow the modules, components, or their functionality or hardware may beimplemented in different embodiments.

EXAMPLES

The following examples pertain to further embodiments.

Example 1 is a system for generating a materialized view over anexternal table. The system includes means for connecting a databaseplatform to an external table such that the database platform has readaccess for the external table and does not have write access for theexternal table. The system includes means for generating a materializedview over the external table. The system includes means for receiving anotification that a modification has been made to the external table,the modification comprising one or more of an insert, a delete, or anupdate. The system includes means for refreshing the materialized viewin response to the external table being modified such that thematerialized view comprises an accurate representation of the externaltable.

Example 2 is a system as in Example 1, wherein the external table isbased on a source directory in a data storage platform, wherein the datastorage platform is separate from the database platform.

Example 3 is a system as in any of Examples 1-2, further comprisingmeans for determining content for the materialized view, wherein themeans for determining the content comprises one or more of: means forreceiving an indication to generate the materialized view with thecontent; means for receiving a query on the external table, the querydirected to the content; or means for determining the content based onone or more rows or partitions of the external table that are frequentlyqueried.

Example 4 is a system as in any of Examples 1-3, wherein the means forreceiving the notification comprises one or more of: means for queryingthe data storage platform to determine whether any modifications havebeen made to the source directory; means for receiving a notificationfrom the data storage platform that a modification has been made to thesource directory; or means for receiving a notification from a clientassociated with the source directory, the notification indicating thatthe modification has been made to the source directory.

Example 5 is a system as in any of Examples 1-4, wherein the means forrefreshing the materialized view comprises means for generating a newmaterialized view and means for removing an original materialized viewthat is now stale with respect to the external table.

Example 6 is a system as in any of Examples 1-5, further comprising:means for receiving a query; means for determining one or morepartitions in the external table that are necessary to respond to thequery by reading metadata for the external table; and means fordetermining whether a corresponding materialized view has been generatedfor at least one of the one or more partitions.

Example 7 is a system as in any of Examples 1-6, further comprising:means for generating a query plan for responding to the query, the queryplan comprising a plurality of discrete subtasks; and means forassigning the plurality of discrete subtasks to one or more nodes of anexecution platform; wherein, in response to there being thecorresponding materialized view for at least one of the one or morepartitions, at least one of the plurality of discrete subtasks comprisesreading the corresponding materialized view; and wherein, in response tothere being no corresponding materialized view for at least one of theone or more partitions, at least one of the plurality of discretesubtasks comprises reading the external table.

Example 8 is a system as in any of Examples 1-7, further comprisingmeans for determining whether the corresponding materialized view isstale with respect to the at least one of the one or more partitions.

Example 9 is a system as in any of Examples 1-8, further comprisingmeans for generating metadata for the materialized view, the metadatacomprising one or more of: an identification of one or more partitionsof the external table that are materialized in the materialized view; aminimum/maximum value pair for each of the one or more partitions thatare materialized in the materialized view; or a timestamp for a mostrecent refresh of the materialized view.

Example 10 is a system as in any of Examples 1-9, further comprising:means for reading a timestamp for a most recent modification to theexternal table for each of the one or more partitions that arematerialized in the materialized view; means for reading the timestampfor the most recent refresh of the materialized view; and means fordetermining whether the materialized view is stale with respect to theexternal table based on the timestamp for the most recent refresh of thematerialized view and each timestamp for the most recent modification tothe external table for each of the one or more partitions that arematerialized in the materialized view.

Example 11 is a method for generating a materialized view over anexternal table. The method includes connecting a database platform to anexternal table such that the database platform has read access for theexternal table and does not have write access for the external table.The method includes generating, by the database platform, a materializedview over the external table. The method includes receiving anotification that a modification has been made to the external table,the modification comprising one or more of an insert, a delete, or anupdate. The method includes, in response to the external table beingmodified, refreshing the materialized view such that the materializedview comprises an accurate representation of the external table.

Example 12 is a method as in Example 11, wherein the external table isbased on a source directory in a data storage platform, wherein the datastorage platform is separate from the database platform.

Example 13 is a method as in any of Examples 11-12, further comprisingdetermining content for the materialized view by one or more of:receiving an indication to generate the materialized view with thecontent; receiving a query on the external table, the query directed tothe content; or determining the content based on one or more rows orpartitions of the external table that are frequently queried.

Example 14 is a method as in any of Examples 11-13, wherein receivingthe notification comprises one or more of: querying the data storageplatform to determine whether any modifications have been made to thesource directory; receiving a notification from the data storageplatform that a modification has been made to the source directory; orreceiving a notification from a client associated with the sourcedirectory, the notification indicating that the modification has beenmade to the source directory.

Example 15 is a method as in any of Examples 11-14, wherein refreshingthe materialized view comprises generating a new materialized view andremoving an original materialized view that is now stale with respect tothe external table.

Example 16 is a method as in any of Examples 11-15, further comprising:receiving a query; determining one or more partitions in the externaltable that are necessary to respond to the query by reading metadata forthe external table; and determining whether a corresponding materializedview has been generated for at least one of the one or more partitions.

Example 17 is a method as in any of Examples 11-16, further comprising:generating a query plan for responding to the query, the query plancomprising a plurality of discrete subtasks; and assigning the pluralityof discrete subtasks to one or more nodes of an execution platform;wherein, in response to there being the corresponding materialized viewfor at least one of the one or more partitions, at least one of theplurality of discrete subtasks comprises reading the correspondingmaterialized view; and wherein, in response to there being nocorresponding materialized view for at least one of the one or morepartitions, at least one of the plurality of discrete subtasks comprisesreading the external table.

Example 18 is a method as in any of Examples 11-17, further comprisingdetermining whether the corresponding materialized view is stale withrespect to the at least one of the one or more partitions.

Example 19 is a method as in any of Examples 11-18, further comprisinggenerating metadata for the materialized view, the metadata comprisingone or more of: an identification of one or more partitions of theexternal table that are materialized in the materialized view; aminimum/maximum value pair for each of the one or more partitions thatare materialized in the materialized view; or a timestamp for a mostrecent refresh of the materialized view.

Example 20 is a method as in any of Examples 11-19, further comprising:reading a timestamp for a most recent modification to the external tablefor each of the one or more partitions that are materialized in thematerialized view; reading the timestamp for the most recent refresh ofthe materialized view; and determining whether the materialized view isstale with respect to the external table based on the timestamp for themost recent refresh of the materialized view and each timestamp for themost recent modification to the external table for each of the one ormore partitions that are materialized in the materialized view.

Example 21 is a processor that is programmable to execute instructionsstored in non-transitory computer readable storage media. Theinstructions include connecting a database platform to an external tablesuch that the database platform has read access for the external tableand does not have write access for the external table. The instructionsinclude generating, by the database platform, a materialized view overthe external table. The instructions include receiving a notificationthat a modification has been made to the external table, themodification comprising one or more of an insert, a delete, or anupdate. The instructions include, in response to the external tablebeing modified, refreshing the materialized view such that thematerialized view comprises an accurate representation of the externaltable. The instructions are such that the external table is based on asource directory in a data storage platform, wherein the data storageplatform is separate from the database platform.

Example 22 is a processor as in Example 21, wherein the instructionsfurther comprise determining content for the materialized view by one ormore of: receiving an indication to generate the materialized view withthe content; receiving a query on the external table, the query directedto the content; or determining the content based on one or more rows orpartitions of the external table that are frequently queried.

Example 23 is a processor as in any of Examples 21-22, wherein receivingthe notification comprises one or more of: querying the data storageplatform to determine whether any modifications have been made to thesource directory; receiving a notification from the data storageplatform that a modification has been made to the source directory; orreceiving a notification from a client associated with the sourcedirectory, the notification indicating that the modification has beenmade to the source directory.

Example 24 is a processor as in any of Examples 21-23, whereinrefreshing the materialized view comprises generating a new materializedview and removing an original materialized view that is now stale withrespect to the external table.

Example 25 is a processor as in any of Examples 21-24, wherein theinstructions further comprise: receiving a query; determining one ormore partitions in the external table that are necessary to respond tothe query by reading metadata for the external table; determiningwhether a corresponding materialized view has been generated for atleast one of the one or more partitions; generating a query plan forresponding to the query, the query plan comprising a plurality ofdiscrete subtasks; and assigning the plurality of discrete subtasks toone or more nodes of an execution platform; wherein, in response tothere being the corresponding materialized view for at least one of theone or more partitions, at least one of the plurality of discretesubtasks comprises reading the corresponding materialized view; andwherein, in response to there being no corresponding materialized viewfor at least one of the one or more partitions, at least one of theplurality of discrete subtasks comprises reading the external table.

Example 26 is a system for querying over an external table. The systemincludes means for connecting a database platform to an external tablesuch that the database platform has read access for the external tableand does not have write access for the external table. The systemincludes means for receiving a query comprising a predicate, the querydirected at least to data in the external table. The system includesmeans for determining, based on metadata, one or more partitions in theexternal table comprising data satisfying the predicate. The systemincludes means for pruning, based on the metadata, all partitions in theexternal table that do not comprise any data satisfying the predicate.The system includes means for generating a query plan comprising aplurality of discrete subtasks. The system includes means for assigning,based on the metadata, the plurality of discrete subtasks to one or morenodes in an execution platform.

Example 27 is a system as in Example 26, wherein the external table isbased on a source directory in a data storage platform, wherein the datastorage platform is separate from the database platform.

Example 28 is a system as in any of Examples 26-27, further comprising:means for determining, based on the metadata, one or more partitions ininternal database data comprising data satisfying the predicate; andmeans for pruning, based on the metadata, all partitions in the internaldatabase data that do not comprise any data satisfying the predicate;wherein the database platform has read access and further has writeaccess to the internal database data.

Example 29 is a system as in any of Examples 26-28, wherein, in responseto there being one or more partitions from each of the external tableand the internal database data that comprise data satisfying thepredicate, the plurality of discrete subtasks of the query plan comprisetasks for processing partitions in the external table and further forprocessing partitions in the internal database data.

Example 30 is a system as in any of Examples 26-29, wherein the meansfor assigning, based on the metadata, the plurality of discrete subtasksfurther comprises: means for reading the metadata to determine whether afirst partition is stored in a cache of any certain node of theexecution platform; and in response to the first partition being storedin the cache of the certain node of the execution platform, means forassigning the first partition to the certain node.

Example 31 is a system as in any of Examples 26-30, further comprisingmeans for generating the metadata, wherein the metadata comprisesinformation about data stored in the external table, the informationcomprising one or more of: cumulative table metadata for the externaltable; a grouping expression property for a grouping of partitions inthe external table; an expression property for a partition of theexternal table; partition statistics for a partition of the externaltable; or a column expression property for a column of a partition ofthe external table.

Example 32 is a system as in any of Examples 26-31, further comprisingmeans for receiving a notification that a modification has been made tothe external table, the modification comprising one or more of aninsert, a delete, or an update.

Example 33 is a system as in any of Examples 26-32, further comprisingmeans for refreshing the metadata in response to the modification beingmade to the external table.

Example 34 is a system as in any of Examples 26-33, further comprisingmeans for refreshing the metadata in response to a threshold number ofmodifications being made to the external table.

Example 35 is a system as in any of Examples 26-34, further comprisingmeans for storing the metadata in a partition storage object on a sharedstorage platform associated with the database platform, wherein theshared storage platform is separate from the external table.

Example 36 a method for querying over an external table. The methodincludes connecting a database platform to an external table such thatthe database platform has read access for the external table and doesnot have write access for the external table. The method includesreceiving a query comprising a predicate, the query directed at least todata in the external table. The method includes determining, based onmetadata, one or more partitions in the external table comprising datasatisfying the predicate. The method includes pruning, based on themetadata, all partitions in the external table that do not comprise anydata satisfying the predicate. The method includes generating a queryplan comprising a plurality of discrete subtasks. The method includesassigning, based on the metadata, the plurality of discrete subtasks toone or more nodes in an execution platform.

Example 37 is a method as in Example 36, wherein the external table isbased on a source directory in a data storage platform, wherein the datastorage platform is separate from the database platform.

Example 38 is a method as in any of Examples 36-37, further comprising:determining, based on the metadata, one or more partitions in internaldatabase data comprising data satisfying the predicate; and pruning,based on the metadata, all partitions in the internal database data thatdo not comprise any data satisfying the predicate; wherein the databaseplatform has read access and further has write access to the internaldatabase data.

Example 39 is a method as in any of Examples 36-38, wherein, in responseto there being one or more partitions from each of the external tableand the internal database data that comprise data satisfying thepredicate, the plurality of discrete subtasks of the query plan comprisetasks for processing partitions in the external table and further forprocessing partitions in the internal database data.

Example 40 is a method as in any of Examples 36-39, wherein assigning,based on the metadata, the plurality of discrete subtasks furthercomprises: reading the metadata to determine whether a first partitionis stored in a cache of any certain node of the execution platform; andin response to the first partition being stored in the cache of thecertain node of the execution platform, assigning the first partition tothe certain node.

Example 41 is a method as in any of Examples 36-40, further comprisinggenerating the metadata, wherein the metadata comprises informationabout data stored in the external table, the information comprising oneor more of: cumulative table metadata for the external table; a groupingexpression property for a grouping of partitions in the external table;an expression property for a partition of the external table; partitionstatistics for a partition of the external table; or a column expressionproperty for a column of a partition of the external table.

Example 42 is a method as in any of Examples 36-41, further comprisingreceiving a notification that a modification has been made to theexternal table, the modification comprising one or more of an insert, adelete, or an update.

Example 43 is a method as in any of Examples 36-42, further comprisingrefreshing the metadata in response to the modification being made tothe external table.

Example 44 is a method as in any of Examples 36-43, further comprisingrefreshing the metadata in response to a threshold number ofmodifications being made to the external table.

Example 45 is a method as in any of Examples 36-44, further comprisingstoring the metadata in a partition storage object on a shared storageplatform associated with the database platform, wherein the sharedstorage platform is separate from the external table.

Example 47 is a processor that is programmable to execute instructionsstored in non-transitory computer readable storage media, theinstructions comprising: connecting a database platform to an externaltable such that the database platform has read access for the externaltable and does not have write access for the external table; receiving aquery comprising a predicate, the query directed at least to data in theexternal table; determining, based on metadata, one or more partitionsin the external table comprising data satisfying the predicate; pruning,based on the metadata, all partitions in the external table that do notcomprise any data satisfying the predicate; generating a query plancomprising a plurality of discrete subtasks; and assigning, based on themetadata, the plurality of discrete subtasks to one or more nodes in anexecution platform; wherein the external table is based on a sourcedirectory in a data storage platform, wherein the data storage platformis separate from the database platform.

Example 48 is a processor as in Example 47, the instructions furthercomprising: determining, based on the metadata, one or more partitionsin internal database data comprising data satisfying the predicate; andpruning, based on the metadata, all partitions in the internal databasedata that do not comprise any data satisfying the predicate; wherein thedatabase platform has read access and further has write access to theinternal database data; and wherein, in response to there being one ormore partitions from each of the external table and the internaldatabase data that comprise data satisfying the predicate, the pluralityof discrete subtasks of the query plan comprise tasks for processingdata partitions in the external table and further for processingpartitions in the internal database data.

Example 49 is a processor as in any of Examples 47-48, whereinassigning, based on the metadata, the plurality of discrete subtasksfurther comprises: reading the metadata to determine whether a firstpartition is stored in a cache of any certain node of the executionplatform; and in response to the first partition being stored in thecache of the certain node of the execution platform, assigning the firstpartition to the certain node.

Example 50 is a processor as in any of Examples 47-49, wherein theinstructions further comprise generating the metadata, wherein themetadata comprises information about data stored in the external table,the information comprising one or more of: cumulative table metadata forthe external table; a grouping expression property for a grouping ofpartitions in the external table; an expression property for a partitionof the external table; partition statistics for a partition of theexternal table; or a column expression property for a column of apartition of the external table.

Example 51 is a processor as in any of Examples 47-50, wherein theinstructions further comprise: receiving a notification that amodification has been made to the external table, the modificationcomprising one or more of an insert, a delete, or an update; andrefreshing the metadata in response to the modification being made tothe external table.

Example 52 is a system. The system includes means for receiving, by adatabase platform, read access to a source directory in a data storageplatform that is separate from the database platform. The systemincludes means for defining an external table based on the sourcedirectory. The system includes means for connecting the databaseplatform to the external table such that the database platform has readaccess for the external table and does not have write access for theexternal table. The system includes means for generating metadata forthe external table, the metadata comprising information about datastored in the external table. The system includes means for receiving anotification that a modification has been made to the source directory,the modification comprising one or more of an insert, a delete, or anupdate. The system includes means for refreshing the metadata for theexternal table in response to the modification being made to the sourcedirectory.

Example 53 is a system as in Example 52, wherein the means forgenerating the metadata comprises one or more of: means for definingcumulative table metadata for the external table; means for defining agrouping expression property for a grouping of partitions in theexternal table; means for defining an expression property for apartition of the external table; means for defining partition statisticsfor a partition of the external table; or means for defining a columnexpression property for a column of a partition of the external table.

Example 54 is a system as in any of Examples 52-53, further comprisingmeans for storing the metadata in a partition storage object on a sharedstorage platform associated with the database platform, wherein theshared storage platform is separate from the data storage platform.

Example 55 is a system as in any of Examples 52-54, wherein the meansfor refreshing the metadata comprises one or more of: means forrefreshing the metadata in response to any modification being made tothe source directory; means for refreshing the metadata at thresholdtime periods; means for refreshing the metadata in response to athreshold number of modifications being made to the source directory; ormeans for refreshing the metadata in response to a client request torefresh the metadata.

Example 56 is a system as in any of Examples 52-55, further comprisingmeans for reading multiple different file formats in the sourcedirectory.

Example 57 is a system as in any of Examples 52-56, wherein the meansfor receiving the notification comprises one or more of: means forquerying the data storage platform to determine whether anymodifications have been made to the source directory; means forreceiving a notification from the data storage platform that amodification has been made to the source directory; or means forreceiving a notification from a client associated with the sourcedirectory, the notification indicating that the modification has beenmade to the source directory.

Example 58 is a system as in any of Examples 52-57, wherein the meansfor defining the external table based on the source directory comprises:means for receiving an indication of a hierarchal structure in thesource directory, the hierarchal structure defining folders andsubfolders for data in the source directory; means for receiving anindication of a partitioning structure for data in the source directory;and means for defining partitions in the external table based on wherefiles are uploaded within the hierarchical structure and further basedon the partitioning structure.

Example 59 is a system as in any of Examples 52-58, wherein the meansfor defining the partitions in the external table comprises means forgenerating metadata for the partitions in the external table and meansfor pointing the metadata to the appropriate folders and subfolders inthe source directory.

Example 60 is a system as in any of Examples 52-59, wherein the meansfor receiving the notification comprises means for receiving anotification that a certain subfolder within the source directory hasbeen modified.

Example 61 is a system as in any of Examples 52-60, further comprising:means for identifying a certain partition in the external table thatcorresponds with the certain subfolder within the source directory; andmeans for generating change tracking metadata for the certain partitionindicating how the certain subfolder was modified and when the certainsubfolder was modified.

Example 62 is a method. The method includes receiving, by a databaseplatform, read access to a source directory in a data storage platformthat is separate from the database platform. The method includesdefining an external table based on the source directory. The methodincludes connecting the database platform to the external table suchthat the database platform has read access for the external table anddoes not have write access for the external table. The method includesgenerating metadata for the external table, the metadata comprisinginformation about data stored in the external table. The method includesreceiving a notification that a modification has been made to the sourcedirectory, the modification comprising one or more of an insert, adelete, or an update. The method includes refreshing the metadata forthe external table in response to the modification being made to thesource directory.

Example 63 is a method as in Example 62, wherein generating the metadatacomprises one or more of: defining cumulative table metadata for theexternal table; defining a grouping expression property for a groupingof partitions in the external table; defining an expression property fora partition of the external table; defining partition statistics for apartition of the external table; or defining a column expressionproperty for a column of a partition of the external table.

Example 64 is a method as in any of Examples 62-63, further comprisingstoring the metadata in a partition storage object on a shared storageplatform associated with the database platform, wherein the sharedstorage platform is separate from the data storage platform.

Example 65 is a method as in any of Examples 62-64, wherein refreshingthe metadata comprises one or more of: refreshing the metadata inresponse to any modification being made to the source directory;refreshing the metadata at threshold time periods; refreshing themetadata in response to a threshold number of modifications being madeto the source directory; or refreshing the metadata in response to aclient request to refresh the metadata.

Example 66 is a method as in any of Examples 62-65, further comprisingreading multiple different file formats in the source directory.

Example 67 is a method as in any of Examples 62-66, wherein receivingthe notification comprises one or more of: querying the data storageplatform to determine whether any modifications have been made to thesource directory; receiving a notification from the data storageplatform that a modification has been made to the source directory; orreceiving a notification from a client associated with the sourcedirectory, the notification indicating that the modification has beenmade to the source directory.

Example 68 is a method as in any of Examples 62-67, wherein defining theexternal table based on the source directory comprises: receiving anindication of a hierarchical structure in the source directory, thehierarchical structure defining folders and subfolders for data in thesource directory; receiving an indication of a partitioning structurefor data in the source directory; and defining partitions in theexternal table based on where files are uploaded within the hierarchicalstructure and further based on the partitioning structure.

Example 69 is a method as in any of Examples 62-68, wherein defining thepartitions in the external table comprises generating metadata for thepartitions in the external table and pointing the metadata to theappropriate folders and subfolders in the source directory.

Example 70 is a method as in any of Examples 62-69, wherein receivingthe notification comprises means for receiving a notification that acertain subfolder within the source directory has been modified.

Example 71 is a method as in any of Examples 62-70, further comprising:identifying a certain partition in the external table that correspondswith the certain subfolder within the source directory; and generatingchange tracking metadata for the certain partition indicating how thecertain subfolder was modified and when the certain subfolder wasmodified.

Example 72 is a processor that is programmable to execute instructionsstored in non-transitory computer readable storage media, theinstructions comprising: receiving, by a database platform, read accessto a source directory in a data storage platform that is separate fromthe database platform; defining an external table based on the sourcedirectory; connecting the database platform to the external table suchthat the database platform has read access for the external table anddoes not have write access for the external table; generating metadatafor the external table, the metadata comprising information about datastored in the external table; receiving a notification that amodification has been made to the source directory, the modificationcomprising one or more of an insert, a delete, or an update; andrefreshing the metadata for the external table in response to themodification being made to the source directory.

Example 73 is a processor as in Example 72, wherein generating themetadata comprises one or more of: defining cumulative table metadatafor the external table; defining a grouping expression property for agrouping of partitions in the external table; defining an expressionproperty for a partition of the external table; defining partitionstatistics for a partition of the external table; or defining a columnexpression property for a column of a partition of the external table.

Example 74 is a processor as in any of Examples 72-73, wherein theinstructions further comprise storing the metadata in a partitionstorage object on a shared storage platform associated with the databaseplatform, wherein the shared storage platform is separate from the datastorage platform.

Example 75 is a processor as in any of Examples 72-74, whereinrefreshing the metadata comprises one or more of: refreshing themetadata in response to any modification being made to the sourcedirectory; refreshing the metadata at threshold time periods; refreshingthe metadata in response to a threshold number of modifications beingmade to the source directory; or refreshing the metadata in response toa client request to refresh the metadata.

Example 76 is a processor as in any of Examples 72-75, wherein definingthe external table based on the source directory comprises: receiving anindication of a hierarchical structure in the source directory, thehierarchical structure defining folders and subfolders for data in thesource directory; receiving an indication of a partitioning structurefor data in the source directory; and defining partitions in theexternal table based on where files are uploaded within the hierarchicalstructure and further based on the partitioning structure.

Various techniques, or certain aspects or portions thereof, may take theform of program code (i.e., instructions) embodied in tangible media,such as floppy diskettes, CD-ROMs, hard drives, a non-transitorycomputer readable storage medium, or any other machine-readable storagemedium wherein, when the program code is loaded into and executed by amachine, such as a computer, the machine becomes an apparatus forpracticing the various techniques. In the case of program code executionon programmable computers, the computing device may include a processor,a storage medium readable by the processor (including volatile andnon-volatile memory and/or storage elements), at least one input device,and at least one output device. The volatile and non-volatile memoryand/or storage elements may be a RAM, an EPROM, a flash drive, anoptical drive, a magnetic hard drive, or another medium for storingelectronic data. One or more programs that may implement or utilize thevarious techniques described herein may use an application programminginterface (API), reusable controls, and the like. Such programs may beimplemented in a high-level procedural, functional, object-orientedprogramming language to communicate with a computer system. However, theprogram(s) may be implemented in assembly or machine language, ifdesired. In any case, the language may be a compiled or interpretedlanguage, and combined with hardware implementations.

It should be understood that many of the functional units described inthis specification may be implemented as one or more components, whichis a term used to more particularly emphasize their implementationindependence. For example, a component may be implemented as a hardwarecircuit comprising custom very large-scale integration (VLSI) circuitsor gate arrays, off-the-shelf semiconductors such as logic chips,transistors, or other discrete components. A component may also beimplemented in programmable hardware devices such as field programmablegate arrays, programmable array logic, programmable logic devices, orthe like.

Components may also be implemented in software for execution by varioustypes of processors. An identified component of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions, which may, for instance, be organized as an object, aprocedure, or a function. Nevertheless, the executables of an identifiedcomponent need not be physically located together but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the component and achieve the statedpurpose for the component.

Indeed, a component of executable code may be a single instruction, ormany instructions, and may even be distributed over several differentcode segments, among different programs, and across several memorydevices. Similarly, operational data may be identified and illustratedherein within components and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork. The components may be passive or active, including agentsoperable to perform desired functions.

Reference throughout this specification to “an example” means that aparticular feature, structure, or characteristic described in connectionwith the example is included in at least one embodiment of the presentdisclosure. Thus, the appearance of the phrase “in an example” invarious places throughout this specification are not necessarily allreferring to the same embodiment.

As used herein, a plurality of items, structural elements, compositionalelements, and/or materials may be presented in a common list forconvenience. However, these lists should be construed as though eachmember of the list is individually identified as a separate and uniquemember. Thus, no individual member of such a list should be construed asa de facto equivalent of any other member of the same list solely basedon its presentation in a common group without indications to thecontrary. In addition, various embodiments and examples of the presentdisclosure may be referred to herein along with alternatives for thevarious components thereof. It is understood that such embodiments,examples, and alternatives are not to be construed as de factoequivalents of one another but are to be considered as separate andautonomous representations of the present disclosure.

Although the foregoing has been described in some detail for purposes ofclarity, it will be apparent that certain changes and modifications maybe made without departing from the principles thereof. It should benoted that there are many alternative ways of implementing both theprocesses and apparatuses described herein. Accordingly, the presentembodiments are to be considered illustrative and not restrictive.

Those having skill in the art will appreciate that many changes may bemade to the details of the above-described embodiments without departingfrom the underlying principles of the disclosure.

1. A system comprising: means for defining, by a database platform, anexternal table based on content that is stored in an external datastorage platform that is separate from the database platform, thedatabase platform not having write access to the external table; meansfor generating metadata for the external table, the metadata comprisinginformation about data stored in the external table; means for receivinga notification that a modification has been made to the content; andmeans for refreshing the metadata for the external table in response tothe modification being made to the content.
 2. The system of claim 1,wherein the means for generating the metadata comprises one or more of:means for defining cumulative table metadata for the external table;means for defining a grouping expression property for a grouping ofpartitions in the external table; means for defining an expressionproperty for a partition of the external table; means for definingpartition statistics for a partition of the external table; and meansfor defining a column expression property for a column of a partition ofthe external table.
 3. The system of claim 1, further comprising meansfor storing the metadata in a partition storage object on a sharedstorage platform associated with the database platform, wherein theshared storage platform is separate from the external data storageplatform.
 4. The system of claim 1, further comprising one or more of:means for refreshing the metadata at threshold time periods; means forrefreshing the metadata in response to a threshold number ofmodifications being made to the content; and means for refreshing themetadata in response to a request to refresh the metadata.
 5. The systemof claim 1, further comprising means for reading multiple different fileformats in the content in the external data storage platform.
 6. Thesystem of claim 1, wherein the means for receiving the notification thatthe modification has been made to the content comprises one or more of:means for querying the external data storage platform to determinewhether any modifications have been made to the content; means forreceiving the notification from the external data storage platform; andmeans for receiving the notification from a client associated with thecontent.
 7. The system of claim 1, wherein the means for defining theexternal table based on the content comprises: means for receiving anindication of a hierarchical structure of the content in the externaldata storage platform, the hierarchical structure defining folders andsubfolders; means for receiving an indication of a partitioningstructure for the content in the external data storage platform; andmeans for defining partitions in the external table based on where filesare uploaded within the hierarchical structure and further based on thepartitioning structure.
 8. The system of claim 7, wherein the means fordefining the partitions in the external table comprises: means forgenerating metadata for the partitions in the external table; and meansfor pointing the metadata to the appropriate folders and subfolders ofthe hierarchical structure.
 9. The system of claim 8, wherein the meansfor receiving the notification comprises means for receiving anotification that a certain subfolder within the hierarchical structurehas been modified.
 10. The system of claim 9, further comprising: meansfor identifying a certain partition in the external table thatcorresponds with the certain subfolder; and means for generating changetracking metadata for the certain partition, the change trackingmetadata indicating how the certain subfolder was modified and when thecertain subfolder was modified.
 11. A method comprising: defining, by adatabase platform, an external table based on content that is stored inan external data storage platform that is separate from the databaseplatform, the database platform not having write access to the externaltable; generating metadata for the external table, the metadatacomprising information about data stored in the external table;receiving a notification that a modification has been made to thecontent; and refreshing the metadata for the external table in responseto the modification being made to the content.
 12. The method of claim11, wherein generating the metadata comprises one or more of: definingcumulative table metadata for the external table; defining a groupingexpression property for a grouping of partitions in the external table;defining an expression property for a partition of the external table;defining partition statistics for a partition of the external table; anddefining a column expression property for a column of a partition of theexternal able.
 13. The method of claim 11, further comprising storingthe metadata in a partition storage object on a shared storage platformassociated with the database platform, wherein the shared storageplatform is separate from the external data storage platform.
 14. Themethod of claim 11, further comprising one or more of: refreshing themetadata at threshold time periods; refreshing the metadata in responseto a threshold number of modifications being made to the content; andrefreshing the metadata in response to a request to refresh themetadata.
 15. The method of claim 11, further comprising readingmultiple different file formats in the content in the external datastorage platform.
 16. The method of claim 11, wherein receiving thenotification that the modification has been made to the contentcomprises one or more of: querying the external data storage platform todetermine whether any modifications have been made to the content;receiving the notification from the external data storage platform; andreceiving the notification from a client associated with the content.17. The method of claim 11, wherein defining the external table based onthe content comprises: receiving an indication of a hierarchicalstructure of the content in the external data storage platform, thehierarchical structure defining folders and subfolders; receiving anindication of a partitioning structure for the content in the externaldata storage platform; and defining partitions in the external tablebased on where files are uploaded within the hierarchical structure andfurther based on the partitioning structure.
 18. The method of claim 17,wherein defining the partitions in the external table comprises:generating metadata for the partitions in the external table; andpointing the metadata to the appropriate folders and subfolders of thehierarchical structure.
 19. The method of claim 18, wherein receivingthe notification comprises receiving a notification that a certainsubfolder within the hierarchical structure has been modified.
 20. Themethod of claim 19, further comprising: identifying a certain partitionin the external table that corresponds with the certain subfolder; andgenerating change tracking metadata for the certain partition, thechange tracking metadata indicating how the certain subfolder wasmodified and when the certain subfolder was modified.
 21. One or morenon-transitory computer readable storage media storing instructionsthat, when executed by one or more processors, cause the one or moreprocessors to perform operations comprising: defining, by a databaseplatform, an external table based on content that is stored in anexternal data storage platform that is separate from the databaseplatform, the database platform not having write access to the externaltable; generating metadata for the external table, the metadatacomprising information about data stored in the external table;receiving a notification that a modification has been made to thecontent; and refreshing the metadata for the external table in responseto the modification being made to the content.
 22. The non-transitorycomputer readable storage media of claim 21, wherein generating themetadata comprises one or more of: defining cumulative table metadatafor the external table; defining a grouping expression property for agrouping of partitions in the external table; defining an expressionproperty for a partition of the external table; defining partitionstatistics for a partition of the external table; and defining a columnexpression property for a column of a partition of the external table.23. The non-transitory computer readable storage media of claim 21,wherein the operations further comprise storing the metadata in apartition storage object on a shared storage platform associated withthe database platform, wherein the shared storage platform is separatefrom the external data storage platform.
 24. The non-transitory computerreadable storage media of claim 21, further comprising one or more of:refreshing the metadata at threshold time periods; refreshing themetadata in response to a threshold number of modifications being madeto the content; and refreshing the metadata in response to a request torefresh the metadata.
 25. The non-transitory computer readable storagemedia of claim 21, wherein defining the external table based on thecontent comprises: receiving an indication of a hierarchical structureof the content in the external data storage platform, the hierarchicalstructure defining folders and subfolders; receiving an indication of apartitioning structure for the content in the external data storageplatform; and defining partitions in the external table based on wherefiles are uploaded within the hierarchical structure and further basedon the partitioning structure.
 26. The non-transitory computer readablestorage media of claim 21, the operations further comprising readingmultiple different file formats in the content in the external datastorage platform.
 27. The non-transitory computer readable storage mediaof claim 21, wherein receiving the notification that the modificationhas been made to the content comprises one or more of: querying theexternal data storage platform to determine whether any modificationshave been made to the content; receiving the notification from theexternal data storage platform; and receiving the notification from aclient associated with the content.
 28. The non-transitory computerreadable storage media of claim 27, wherein defining the partitions inthe external table comprises: generating metadata for the partitions inthe external table; and pointing the metadata to the appropriate foldersand subfolders of the hierarchical structure.
 29. The non-transitorycomputer readable storage media of claim 28, wherein receiving thenotification comprises receiving a notification that a certain subfolderwithin the hierarchical structure has been modified.
 30. Thenon-transitory computer readable storage media of claim 29, furthercomprising: identifying a certain partition in the external table thatcorresponds with the certain subfolder; and generating change trackingmetadata for the certain partition, the change tracking metadataindicating how the certain subfolder was modified and when the certainsubfolder as modified.
 31. A database platform comprising: at least oneprocessor; and one or more non-transitory computer readable storagemedia containing instructions executable by the at least one processorfor causing the at least one processor to perform operations comprising:defining an external table based on content that is stored in anexternal data storage platform that is separate from the databaseplatform, the database platform not having write access to the externaltable; generating metadata for the external table, the metadatacomprising information about data stored in the external table;receiving a notification that a modification has been made to thecontent; and refreshing the metadata for the external table in responseto the modification being made to the content.
 32. The database platformof claim 31, wherein generating the metadata comprises one or more of:defining cumulative table metadata for the external table; defining agrouping expression property for a grouping of partitions in theexternal table; defining an expression property for a partition of theexternal table; defining partition statistics for a partition of theexternal table; and defining a column expression property for a columnof a partition of the external table.
 33. The database platform of claim31, further comprising storing the metadata in a partition storageobject on a shared storage platform associated with the databaseplatform, wherein the shared storage platform is separate from theexternal data storage platform.
 34. The database platform of claim 31,further comprising one or more of: refreshing the metadata at thresholdtime periods; refreshing the metadata in response to a threshold numberof modifications being made to the content; and refreshing the metadatain response to a request to refresh the metadata.
 35. The databaseplatform of claim 31, further comprising reading multiple different fileformats in the content in the external data storage platform.
 36. Thedatabase platform of claim 31, wherein receiving the notification thatthe modification has been made to the content comprises one or more of:querying the external data storage platform to determine whether anymodifications have been made to the content; receiving the notificationfrom the external data storage platform; and receiving the notificationfrom a client associated with the content.
 37. The database platform ofclaim 31, wherein defining the external table based on the contentcomprises: receiving an indication of a hierarchical structure of thecontent in the external data storage platform, the hierarchicalstructure defining folders and subfolders; receiving an indication of apartitioning structure for the content in the external data storageplatform; and defining partitions in the external table based on wherefiles are uploaded within the hierarchical structure and further basedon the partitioning structure.
 38. The database platform of claim 37,wherein defining the partitions in the external table comprises:generating metadata for the partitions in the external table; andpointing the metadata to the appropriate folders and subfolders of thehierarchical structure.
 39. The database platform of claim 38, whereinreceiving the notification comprises receiving a notification that acertain subfolder within the hierarchical structure has been modified.40. The database platform of claim 39, further comprising: identifying acertain partition in the external table that corresponds with thecertain subfolder; and generating change tracking metadata for thecertain partition, the change tracking metadata indicating how thecertain subfolder was modified and when the certain subfolder wasmodified.
 41. The database platform of claim 31, the modificationcomprising one or more of an addition of a file, a deletion of a file,and an update to a file in a source location of the external table onthe external data storage platform.
 42. The system of claim 1, themodification comprising one or more of an addition of a file, a deletionof a file, and an update to a file in a source location of the externaltable on the external data storage platform.
 43. The method of claim 11,the modification comprising one or more of an addition of a file, adeletion of a file, and an update to a file in a source location of theexternal table on the external data storage platform.
 44. Thenon-transitory computer readable storage media of claim 21, themodification comprising one or more of an addition of a file, a deletionof a file, and an update to a file in a source location of the externaltable on the external data storage platform.